Nov 02 07:21:32.105295 ip-10-0-134-79 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 07:21:32.105309 ip-10-0-134-79 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 07:21:32.105319 ip-10-0-134-79 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 07:21:32.105744 ip-10-0-134-79 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 07:21:41.674741 ip-10-0-134-79 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 07:21:41.674754 ip-10-0-134-79 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 81f6a6768870443f96a5912e855b335d --
Nov 02 07:23:28.921161 ip-10-0-134-79 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 07:23:29.302750 ip-10-0-134-79 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 07:23:29.302750 ip-10-0-134-79 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 07:23:29.302750 ip-10-0-134-79 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 07:23:29.302750 ip-10-0-134-79 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 07:23:29.307645 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307565    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:29.307645 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307645    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:29.307645 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307649    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307652    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307655    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307658    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307660    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307663    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307666    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307669    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307671    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307674    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307678    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307681    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307683    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307691    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307694    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307697    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307700    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307702    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307705    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307707    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:29.307770 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307710    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307712    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307715    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307718    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307720    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307723    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307725    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307728    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307730    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307733    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307736    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307738    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307741    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307743    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307747    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307749    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307752    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307756    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307758    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307761    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:29.308243 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307764    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307766    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307768    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307771    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307774    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307776    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307779    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307782    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307784    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307787    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307790    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307792    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307795    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307797    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307800    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307803    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307806    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307808    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307811    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307814    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:29.308718 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307816    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307818    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307822    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307824    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307827    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307830    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307833    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307836    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307839    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307841    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307845    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307848    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307850    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307853    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307855    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307858    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307861    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307863    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307866    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:29.309187 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307868    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307871    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307873    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307876    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307878    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307881    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307883    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307900    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307905    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307909    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307914    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307917    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307921    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307924    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307927    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307930    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307933    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307937    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:29.309648 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307940    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307943    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307945    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307948    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307951    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307954    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.307957    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308734    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308741    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308745    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308747    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308750    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308753    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308757    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308761    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308764    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308767    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308770    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:29.310087 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308773    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308776    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308779    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308781    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308784    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308787    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308789    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308792    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308795    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308798    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308800    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308803    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308806    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308808    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308810    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308813    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308816    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308818    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308821    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308823    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:29.310555 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308826    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308829    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308831    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308834    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308837    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308839    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308842    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308844    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308847    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308850    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308852    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308855    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308858    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308862    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308865    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308868    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308870    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308873    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308876    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308878    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:29.311050 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308882    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308884    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308903    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308906    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308909    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308912    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308914    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308917    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308919    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308922    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308924    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308927    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308929    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308932    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308935    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308938    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308941    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308943    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308946    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308949    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:29.311540 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308952    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308954    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308957    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308960    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308964    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308968    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308971    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308974    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308976    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308979    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308983    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308986    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308988    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308991    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308993    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308996    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.308998    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309001    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309003    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:29.312031 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309006    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309009    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309011    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309014    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309016    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309019    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309021    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309024    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309027    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309030    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309032    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309035    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309038    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309040    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309043    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309046    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309530    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309535    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309539    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:29.312488 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309541    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309544    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309546    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309549    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309554    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309557    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309560    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309562    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309565    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309567    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309570    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309573    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309577    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309581    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309585    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309589    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309592    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309595    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309597    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:29.312941 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309600    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309602    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309606    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309608    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309611    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309614    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309616    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309619    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309621    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309624    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309626    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309629    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309632    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309635    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309638    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309641    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309643    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309647    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309650    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:29.313411 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309653    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309655    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309658    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309661    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309663    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309666    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309668    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309671    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309673    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309692    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309696    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309700    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309703    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309706    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309709    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309711    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309722    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309725    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309728    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309730    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:29.313938 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309733    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309736    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309738    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309741    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309743    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309746    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309748    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309751    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309754    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309756    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309759    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309762    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309765    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309767    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309769    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309772    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309774    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309777    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309779    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309782    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:29.314475 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309784    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309787    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309790    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309792    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309795    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309797    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309800    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309802    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309804    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309810    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309813    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309815    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309818    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309821    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309824    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309828    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309831    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309835    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309837    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:29.315016 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309841    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309843    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309846    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309848    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309851    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.309854    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311041    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311052    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311058    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311063    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311068    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311071    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311075    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311079    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311083    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311086    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311090    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311093    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311096    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311099    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311102    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311105    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311108    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311111    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 07:23:29.315510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311116    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311121    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311125    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311128    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311132    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311136    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311139    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311142    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311145    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311148    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311151    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311154    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311158    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311161    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311165    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311168    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311171    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311174    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311177    2576 flags.go:64] FLAG: --enable-server="true"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311180    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311188    2576 flags.go:64] FLAG: --event-burst="100"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311191    2576 flags.go:64] FLAG: --event-qps="50"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311194    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311197    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 07:23:29.316207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311200    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311204    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311207    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311210    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311213    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311217    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311220    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311223    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311233    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311238    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311241    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311244    2576 flags.go:64] FLAG: --feature-gates=""
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311249    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311251    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311255    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311258    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311261    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311271    2576 flags.go:64] FLAG: --help="false"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311275    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311278    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311281    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311284    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311288    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 07:23:29.316804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311292    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311294    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311297    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311300    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311303    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311306    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311309    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311312    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311315    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311318    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311321    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311324    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311327    2576 flags.go:64] FLAG: --lock-file=""
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311330    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311333    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311336    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311341    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311344    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311354    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311359    2576 flags.go:64] FLAG: --logging-format="text"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311362    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311366    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311369    2576 flags.go:64] FLAG: --manifest-url=""
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311372    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311376    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 07:23:29.317360 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311379    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311383    2576 flags.go:64] FLAG: --max-pods="110"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311386    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311390    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311393    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311396    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311399    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311402    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311405    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311412    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311415    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311417    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311420    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311423    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311426    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311429    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311432    2576 flags.go:64] FLAG: --port="10250"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311435    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311438    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0166363acc16366c0"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311441    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311444    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311447    2576 flags.go:64] FLAG: --register-node="true"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311450    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311454    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311457    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311462    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 07:23:29.318002 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311474    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311481    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311484    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311487    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311490    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311493    2576 flags.go:64] FLAG: --runonce="false"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311496    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311499    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311502    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311506    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311509    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311512    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311515    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311518    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311522    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311525    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311528    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311531    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311534    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311537    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311540    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311545    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311549    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311551    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311560    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 02 07:23:29.318624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311563    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311566    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311569    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311572    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311575    2576 flags.go:64] FLAG: --v="2"
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311579    2576 flags.go:64] FLAG: --version="false"
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311585    2576 flags.go:64] FLAG: --vmodule=""
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311591    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.311600    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311753    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311757    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311760    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311762    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311765    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311767    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311770    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311773    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311775    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311778    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311780    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311783    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311786    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:29.319273 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311788    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311791    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311793    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311796    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311799    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311802    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311804    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311807    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311809    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311812    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311814    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311817    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311820    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311822    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311825    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311827    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311831    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311836    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311838    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311843    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:29.319797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311845    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311848    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311850    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311853    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311856    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311858    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311861    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311864    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311866    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311869    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311872    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311874    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311877    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311879    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311882    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311884    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311899    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311902    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311904    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311907    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:29.320284 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311909    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311912    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311915    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311917    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311920    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311923    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311925    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311928    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311932    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311936    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311939    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311941    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311945    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311947    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311950    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311953    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311956    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311959    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311961    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:29.320758 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311964    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311966    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311969    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311972    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311974    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311977    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311979    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311982    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311985    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311987    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311990    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311992    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311996    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.311999    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312002    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312005    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312008    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312010    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312013    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:29.321225 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312016    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312020    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312025    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312029    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312033    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312037    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312040    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312044    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312047    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312050    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312053    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312055    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312058    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312061    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312063    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:29.321675 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.312091    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312146    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312150    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312153    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312156    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312159    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312162    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312165    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312167    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312170    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312173    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312176    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312178    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312181    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312183    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312186    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312189    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312191    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312194    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312196    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:29.322045 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312201    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312203    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312206    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312208    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312211    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312214    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312217    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312219    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312222    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312224    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312227    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312229    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312232    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312235    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312237    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312240    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312243    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312245    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312248    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312251    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:29.322527 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312253    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312256    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312258    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312260    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312263    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312265    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312268    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312270    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312273    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312275    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312278    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312280    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312285    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312287    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312290    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312293    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312295    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312298    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312301    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:29.323210 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312304    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312307    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312310    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312312    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312315    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312317    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312319    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312322    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312325    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312327    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312330    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312332    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312335    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312337    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312340    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312342    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312345    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312347    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312350    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:29.323915 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312352    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312356    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312359    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312362    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312364    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312367    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312371    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312373    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312376    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312380    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312384    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312387    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312390    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312393    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312396    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312398    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312401    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312403    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312405    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:29.324366 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312408    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312410    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312414    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312416    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312419    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312421    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312424    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312426    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312429    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.312431    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.319186    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.319203    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.319219    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.319224    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319287    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319293    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319297    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319300    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319303    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319306    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:29.324806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319308    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319311    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319314    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319317    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319320    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319322    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319325    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319328    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319331    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319334    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319337    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319339    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319342    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319344    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319347    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319351    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319355    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319358    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319362    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:29.325351 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319365    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319368    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319371    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319375    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319379    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319382    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319385    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319388    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319390    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319393    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319396    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319399    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319401    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319404    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319407    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319409    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319412    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319415    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319418    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319420    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:29.325806 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319423    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319426    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319428    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319431    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319434    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319437    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319440    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319443    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319445    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319448    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319450    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319453    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319456    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319458    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319461    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319463    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319466    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319469    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319472    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:29.326291 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319475    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319477    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319480    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319483    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319485    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319488    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319491    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319493    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319496    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319499    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319501    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319504    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319506    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319509    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319511    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319514    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319517    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319519    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319522    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:29.326757 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319525    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319528    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319530    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319533    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319537    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319541    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319544    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319547    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319550    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319553    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319556    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319559    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319562    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319565    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319567    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319570    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319573    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319575    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319578    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:29.327216 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319581    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319583    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319586    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319589    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.319618    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319759    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319764    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319767    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319770    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319772    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319775    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319778    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319781    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319784    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319786    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319789    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:29.327676 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319792    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319795    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319797    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319800    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319803    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319805    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319808    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319811    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319814    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319817    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319820    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319823    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319825    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319828    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319830    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319834    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319836    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319839    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319842    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:29.328065 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319844    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319846    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319849    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319853    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319857    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319862    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319867    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319872    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319877    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319882    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319903    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319908    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319912    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319915    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319917    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319920    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319923    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319925    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319928    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:29.328537 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319930    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319933    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319935    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319938    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319941    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319944    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319947    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319949    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319952    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319954    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319957    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319959    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319962    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319964    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319967    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319970    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319972    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319974    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319977    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319979    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:29.328998 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319982    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319985    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319987    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319989    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319992    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319994    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319997    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.319999    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320002    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320005    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320007    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320010    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320012    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320015    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320017    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320020    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320023    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320025    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320028    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320031    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:29.329499 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320033    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320036    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320039    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320041    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320044    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320047    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320049    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320051    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320054    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320057    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320059    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320062    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320064    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320067    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320069    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320071    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:29.329980 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320074    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.320098    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320133    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320137    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320140    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320142    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320145    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320147    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320150    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320153    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320156    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320158    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320161    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320163    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320165    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320168    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:23:29.330347 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320171    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320174    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320177    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320179    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320182    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320185    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320187    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320189    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320192    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320194    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320197    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320199    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320202    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320204    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320207    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320209    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320212    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320214    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320217    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:23:29.330797 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320219    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320222    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320224    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320228    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320230    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320232    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320235    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320256    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320260    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320263    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320266    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320269    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320272    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320276    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320281    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320284    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320288    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320291    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320293    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:23:29.331261 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320296    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320299    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320301    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320304    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320307    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320309    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320312    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320314    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320317    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320319    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320322    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320324    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320327    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320330    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320333    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320335    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320338    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320340    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320343    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320345    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:23:29.331746 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320348    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320350    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320352    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320355    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320358    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320360    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320363    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320366    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320369    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320371    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320374    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320376    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320379    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320381    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320384    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320386    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320389    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320391    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320394    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320396    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:23:29.332332 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320399    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320403    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320405    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320408    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320410    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320413    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320415    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320420    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320423    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320425    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320430    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320432    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320435    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:29.320437    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.320494    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.326332    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.327198    2576 server.go:1014] "Starting client certificate rotation"
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.327289    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 07:23:29.332792 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.327319    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 07:23:29.333398 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.329315    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 07:23:29.333398 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.330465    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 07:23:29.340929 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.340913    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 02 07:23:29.344420 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.344406    2576 log.go:25] "Validated CRI v1 image API"
Nov 02 07:23:29.346577 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.346549    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 07:23:29.349706 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.349679    2576 fs.go:135] Filesystem UUIDs: map[13163f21-dcf2-446b-acc4-096087adf2ea:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c5bf0693-a6c6-4492-ae53-04d08eb06a4e:/dev/nvme0n1p3]
Nov 02 07:23:29.349772 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.349705    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 07:23:29.356736 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.356624    2576 manager.go:217] Machine: {Timestamp:2026-11-02 07:23:29.35477585 +0000 UTC m=+0.323368089 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102151 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b7a059e8f80beaee5e77160f981d6 SystemUUID:ec2b7a05-9e8f-80be-aee5-e77160f981d6 BootID:81f6a676-8870-443f-96a5-912e855b335d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d9:19:52:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d9:19:52:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:8f:b5:e8:ba:f4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 07:23:29.356736 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.356733    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 07:23:29.356838 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.356811    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 07:23:29.359625 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.359593    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 07:23:29.359751 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.359627    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-79.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 07:23:29.359798 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.359753    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 07:23:29.359798 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.359761    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 07:23:29.359798 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.359791    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 07:23:29.359877 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.359808    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 07:23:29.360884 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.360874    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 07:23:29.365128 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.365116    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 07:23:29.365242 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.365233    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 07:23:29.365979 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.365969    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 07:23:29.366012 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.365983    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 07:23:29.366012 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.365997    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 07:23:29.366012 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.366006    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 02 07:23:29.366083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.366014    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 07:23:29.366445 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.366433    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 07:23:29.368624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.368606    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 07:23:29.375770 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.375753    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 07:23:29.376595 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.376583    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 07:23:29.376638 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.376614    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 07:23:29.376783 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.376772    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 07:23:29.376818 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.376789    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 07:23:29.376818 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.376796    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 07:23:29.376818 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.376802    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 07:23:29.376818 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.376808    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 07:23:29.376818 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.376814    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 07:23:29.377296 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.377288    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 07:23:29.377330 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.377297    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 07:23:29.377330 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.377304    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 07:23:29.377330 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.377309    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 07:23:29.377330 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.377317    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 07:23:29.377330 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.377326    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 07:23:29.377979 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.377969    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 07:23:29.378010 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.377981    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 07:23:29.381786 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.381771    2576 server.go:1263] "Started kubelet"
Nov 02 07:23:29.381878 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.381851    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 07:23:29.381956 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.381861    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 07:23:29.381956 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.381939    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 07:23:29.382576 ip-10-0-134-79 systemd[1]: Started Kubernetes Kubelet.
Nov 02 07:23:29.383027 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.383012    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 07:23:29.383232 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.383211    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 07:23:29.388044 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.388022    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 07:23:29.389311 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.389288    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 07:23:29.390427 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.390411    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 07:23:29.390587 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.390558    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:29.391121 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.391103    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 07:23:29.391121 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.391121    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 07:23:29.391272 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.391102    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 07:23:29.391272 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.391220    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 07:23:29.391272 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.391230    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 07:23:29.391521 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.391508    2576 factory.go:153] Registering CRI-O factory
Nov 02 07:23:29.391585 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.391527    2576 factory.go:223] Registration of the crio container factory successfully
Nov 02 07:23:29.391625 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.391601    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 07:23:29.391625 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.391611    2576 factory.go:55] Registering systemd factory
Nov 02 07:23:29.391625 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.391618    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 02 07:23:29.391709 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.391651    2576 factory.go:103] Registering Raw factory
Nov 02 07:23:29.391709 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.391666    2576 manager.go:1199] Started watching for new ooms in manager
Nov 02 07:23:29.392706 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.392692    2576 manager.go:319] Starting recovery of all containers
Nov 02 07:23:29.402992 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.402968    2576 manager.go:324] Recovery completed
Nov 02 07:23:29.409597 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.409580    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:29.411273 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.411249    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:29.411359 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.411289    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:29.411359 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.411305    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:29.411853 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.411841    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 02 07:23:29.411853 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.411849    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:29.411984 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.411852    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 07:23:29.411984 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.411902    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 07:23:29.412629 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.412610    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:29.412694 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.412643    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:29.412694 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.412657    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:29.414136 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.414118    2576 policy_none.go:50] "Start"
Nov 02 07:23:29.414178 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.414133    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 07:23:29.414178 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.414150    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 07:23:29.414975 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.413992    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-79.ec2.internal.18e4298687d8105c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-79.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-79.ec2.internal,},FirstTimestamp:2026-11-02 07:23:29.381740636 +0000 UTC m=+0.350332874,LastTimestamp:2026-11-02 07:23:29.381740636 +0000 UTC m=+0.350332874,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-79.ec2.internal,}"
Nov 02 07:23:29.416295 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.416282    2576 policy_none.go:44] "Start"
Nov 02 07:23:29.428428 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.428405    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 07:23:29.430684 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.430649    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 07:23:29.430684 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.430656    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 07:23:29.430831 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.430710    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-79.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 07:23:29.430831 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.430667    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-79.ec2.internal.18e42986899ab13f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-79.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-79.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-79.ec2.internal,},FirstTimestamp:2026-11-02 07:23:29.411273023 +0000 UTC m=+0.379865261,LastTimestamp:2026-11-02 07:23:29.411273023 +0000 UTC m=+0.379865261,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-79.ec2.internal,}"
Nov 02 07:23:29.430831 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.430776    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 07:23:29.442512 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.442481    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 07:23:29.448091 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.448078    2576 manager.go:341] "Starting Device Plugin manager"
Nov 02 07:23:29.448154 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.448102    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 07:23:29.448154 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.448111    2576 server.go:87] "Starting device plugin registration server"
Nov 02 07:23:29.448314 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.448299    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 07:23:29.448377 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.448316    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 07:23:29.448496 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.448474    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 07:23:29.448617 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.448597    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 07:23:29.448617 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.448609    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 07:23:29.449031 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.449014    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 07:23:29.449099 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.449064    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:29.457551 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.457532    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 07:23:29.457640 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.457572    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-79.ec2.internal.18e42986899b0f8c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-79.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-79.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-79.ec2.internal,},FirstTimestamp:2026-11-02 07:23:29.411297164 +0000 UTC m=+0.379889402,LastTimestamp:2026-11-02 07:23:29.411297164 +0000 UTC m=+0.379889402,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-79.ec2.internal,}"
Nov 02 07:23:29.460590 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.460571    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lxvxp"
Nov 02 07:23:29.470309 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.470293    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lxvxp"
Nov 02 07:23:29.478006 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.477940    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-79.ec2.internal.18e42986899b4643  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-79.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-79.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-79.ec2.internal,},FirstTimestamp:2026-11-02 07:23:29.411311171 +0000 UTC m=+0.379903410,LastTimestamp:2026-11-02 07:23:29.411311171 +0000 UTC m=+0.379903410,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-79.ec2.internal,}"
Nov 02 07:23:29.503150 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.503079    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-79.ec2.internal.18e42986899ab13f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-79.ec2.internal.18e42986899ab13f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-79.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-79.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-79.ec2.internal,},FirstTimestamp:2026-11-02 07:23:29.411273023 +0000 UTC m=+0.379865261,LastTimestamp:2026-11-02 07:23:29.412627957 +0000 UTC m=+0.381220196,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-79.ec2.internal,}"
Nov 02 07:23:29.524559 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.524467    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-79.ec2.internal.18e42986899b0f8c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-79.ec2.internal.18e42986899b0f8c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-79.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-79.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-79.ec2.internal,},FirstTimestamp:2026-11-02 07:23:29.411297164 +0000 UTC m=+0.379889402,LastTimestamp:2026-11-02 07:23:29.412649237 +0000 UTC m=+0.381241476,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-79.ec2.internal,}"
Nov 02 07:23:29.549229 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.549209    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:29.550207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.550190    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:29.550272 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.550227    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:29.550272 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.550241    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:29.550272 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.550270    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.552155 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.552065    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-79.ec2.internal.18e42986899b4643\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-79.ec2.internal.18e42986899b4643  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-79.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-79.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-79.ec2.internal,},FirstTimestamp:2026-11-02 07:23:29.411311171 +0000 UTC m=+0.379903410,LastTimestamp:2026-11-02 07:23:29.412661942 +0000 UTC m=+0.381254180,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-79.ec2.internal,}"
Nov 02 07:23:29.588005 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.574787    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-79.ec2.internal.18e429868ba0fd57  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-79.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-134-79.ec2.internal,},FirstTimestamp:2026-11-02 07:23:29.445240151 +0000 UTC m=+0.413832393,LastTimestamp:2026-11-02 07:23:29.445240151 +0000 UTC m=+0.413832393,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-79.ec2.internal,}"
Nov 02 07:23:29.588005 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.584288    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 07:23:29.588005 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.585410    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 07:23:29.588005 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.585430    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 07:23:29.588005 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.585449    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 07:23:29.588005 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.585484    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 07:23:29.607520 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.607494    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-134-79.ec2.internal\" not found" node="ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.607715 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.607700    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:23:29.663749 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.663721    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-79.ec2.internal\" not found" node="ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.685813 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.685779    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"]
Nov 02 07:23:29.685979 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.685848    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:29.686822 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.686805    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:29.686908 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.686833    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:29.686908 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.686843    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:29.689066 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.689052    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:29.689229 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.689216    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.689294 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.689246    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:29.689812 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.689794    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:29.689812 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.689805    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:29.689967 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.689821    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:29.689967 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.689835    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:29.689967 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.689825    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:29.689967 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.689910    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:29.691986 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.691970    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.692060 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.691997    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:29.692495 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.692474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0720e6d8914c83a975428ab7b77292e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" (UID: \"0720e6d8914c83a975428ab7b77292e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.692576 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.692528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04aaad1dd656465cdf1b219bba0dba7b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-79.ec2.internal\" (UID: \"04aaad1dd656465cdf1b219bba0dba7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.692576 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.692567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0720e6d8914c83a975428ab7b77292e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" (UID: \"0720e6d8914c83a975428ab7b77292e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.692726 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.692708    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:29.692765 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.692736    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:29.692765 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.692750    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:29.709373 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.709351    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-79.ec2.internal\" not found" node="ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.711333 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.711318    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-79.ec2.internal\" not found" node="ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.792780 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.792751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0720e6d8914c83a975428ab7b77292e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" (UID: \"0720e6d8914c83a975428ab7b77292e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.792780 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.792781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0720e6d8914c83a975428ab7b77292e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" (UID: \"0720e6d8914c83a975428ab7b77292e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.792953 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.792798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04aaad1dd656465cdf1b219bba0dba7b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-79.ec2.internal\" (UID: \"04aaad1dd656465cdf1b219bba0dba7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.792953 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.792825    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04aaad1dd656465cdf1b219bba0dba7b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-79.ec2.internal\" (UID: \"04aaad1dd656465cdf1b219bba0dba7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.792953 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.792845    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0720e6d8914c83a975428ab7b77292e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" (UID: \"0720e6d8914c83a975428ab7b77292e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.792953 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.792846    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0720e6d8914c83a975428ab7b77292e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" (UID: \"0720e6d8914c83a975428ab7b77292e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.807776 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.807755    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:29.808762 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.808737    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:29.808836 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.808767    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:29.808836 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.808781    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:29.808836 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.808813    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.829651 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:29.829632    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-79.ec2.internal"
Nov 02 07:23:29.829702 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.829654    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-79.ec2.internal\": node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:29.875231 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.875169    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:29.976260 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:29.976223    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:30.013920 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:30.013876    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:30.016050 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:30.016032    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:30.077231 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:30.077055    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:30.181357 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:30.181300    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:30.281462 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:30.281424    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:30.327650 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:30.327626    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 07:23:30.328175 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:30.327779    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 07:23:30.382619 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:30.382590    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:30.483587 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:30.483552    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:30.561637 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:30.561596    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 07:18:29 +0000 UTC" deadline="2028-06-28 13:53:14.230388017 +0000 UTC"
Nov 02 07:23:30.561637 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:30.561626    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14502h29m43.668765144s"
Nov 02 07:23:30.580906 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:30.580870    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:23:30.583587 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:30.583563    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:23:30.583732 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:30.583605    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:23:30.583732 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:30.583621    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:23:30.583732 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:30.583632    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:30.604307 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:30.604273    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0720e6d8914c83a975428ab7b77292e6.slice/crio-c0403c57e79b789165972e9076d6419b76ed0168408c4779079dc501d6300472 WatchSource:0}: Error finding container c0403c57e79b789165972e9076d6419b76ed0168408c4779079dc501d6300472: Status 404 returned error can't find the container with id c0403c57e79b789165972e9076d6419b76ed0168408c4779079dc501d6300472
Nov 02 07:23:30.604650 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:30.604625    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04aaad1dd656465cdf1b219bba0dba7b.slice/crio-e3d627c6e8421ae277ab8773485b04ecbef13520834f978ec6dcc4191da38a1a WatchSource:0}: Error finding container e3d627c6e8421ae277ab8773485b04ecbef13520834f978ec6dcc4191da38a1a: Status 404 returned error can't find the container with id e3d627c6e8421ae277ab8773485b04ecbef13520834f978ec6dcc4191da38a1a
Nov 02 07:23:30.610986 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:30.610962    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 07:23:30.684771 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:30.684736    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:30.791439 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:30.791372    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:30.891676 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:30.891641    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:30.992745 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:30.992711    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:31.088738 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.088666    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:23:31.092828 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.092792    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:31.193572 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.193536    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:31.294449 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.294414    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:23:31.344858 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.344767    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:23:31.344858 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.344784    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:23:31.369038 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.369013    2576 apiserver.go:52] "Watching apiserver"
Nov 02 07:23:31.390508 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.390479    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 07:23:31.391058 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.391035    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:31.398854 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.398832    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 07:23:31.399192 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.399168    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf","openshift-cluster-node-tuning-operator/tuned-rcfgw","openshift-image-registry/node-ca-hd4ds","openshift-network-operator/iptables-alerter-zdfn6","kube-system/global-pull-secret-syncer-rs7kb","openshift-multus/network-metrics-daemon-7j98p","openshift-dns/node-resolver-k4jxw","openshift-multus/multus-additional-cni-plugins-g7x6c","openshift-multus/multus-bp5ct","openshift-network-diagnostics/network-check-target-g88mp","openshift-ovn-kubernetes/ovnkube-node-lg4p5","kube-system/konnectivity-agent-9t5tl"]
Nov 02 07:23:31.401861 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.401840    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.405155 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.405133    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.407651 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.407630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:23:31.412063 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.411419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:23:31.413734 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.413714    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:31.413831 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.413803    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:31.416216 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.416193    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:31.416300 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.416264    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:31.419043 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.418658    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:23:31.420078 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.420053    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 07:23:31.420175 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.420148    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-szw6v\""
Nov 02 07:23:31.420175 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.420148    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 07:23:31.420529 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.420510    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:23:31.420603 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.420511    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 07:23:31.420603 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.420534    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 07:23:31.420707 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.420614    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:23:31.421475 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.421204    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.421576 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.421479    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w2bbn\""
Nov 02 07:23:31.421576 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.421536    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 07:23:31.422692 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.422670    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 07:23:31.422793 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.422748    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 07:23:31.425113 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.425091    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.428424 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.428401    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:31.428534 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.428479    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:31.428914 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.428882    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 07:23:31.429271 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.429242    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 07:23:31.429385 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.429370    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:23:31.429510 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.429494    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c9gk6\""
Nov 02 07:23:31.429575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.429546    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 07:23:31.429634 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.429618    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6xxpk\""
Nov 02 07:23:31.430858 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.430830    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.433281 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.433256    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"]
Nov 02 07:23:31.433385 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.433366    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:23:31.444587 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.444569    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"]
Nov 02 07:23:31.445349 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.445330    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 07:23:31.478461 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.478436    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 07:23:31.478574 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.478508    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-llvxn"
Nov 02 07:23:31.493162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.493142    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 07:23:31.502568 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.502541    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zprsb\" (UniqueName: \"kubernetes.io/projected/948eb959-3a16-4ea4-a87b-9497f8bf46d9-kube-api-access-zprsb\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.502568 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.502568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-cni-multus\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.502744 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.502592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-sys-fs\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.502744 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.502622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8jc5\" (UniqueName: \"kubernetes.io/projected/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-kube-api-access-v8jc5\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.502744 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.502687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-modprobe-d\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.502911 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.502742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxct4\" (UniqueName: \"kubernetes.io/projected/0059120b-d9ca-462c-a74b-bb99ea44c3d7-kube-api-access-gxct4\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:31.502911 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.502792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-cni-bin\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.502911 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.502840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovnkube-script-lib\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.502911 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.502908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-netns\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.503136 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.502954    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3-konnectivity-ca\") pod \"konnectivity-agent-9t5tl\" (UID: \"4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3\") " pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:23:31.503136 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-device-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.503136 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-node-log\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.503136 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6da9cb8-1262-4288-a7fb-5991fff5c41a-host\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:23:31.503324 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8qvr\" (UniqueName: \"kubernetes.io/projected/539550af-7526-447a-b6e3-d268f758aeb1-kube-api-access-d8qvr\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:23:31.503324 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-conf-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.503324 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503241    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-daemon-config\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.503324 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5bj8\" (UniqueName: \"kubernetes.io/projected/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-kube-api-access-j5bj8\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.503504 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-systemd-units\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.503504 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-ovn\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.503504 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:31.503504 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503411    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-k8s-cni-cncf-io\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.503504 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503447    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.503504 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-socket-dir-parent\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.503779 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-etc-kubernetes\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.503779 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503564    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysconfig\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.503779 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-host\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.503779 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-systemd\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.503779 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503700    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.503779 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-kubernetes\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.503779 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503741    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-systemd\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.503779 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-tmp\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.503779 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crjhf\" (UniqueName: \"kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf\") pod \"network-check-target-g88mp\" (UID: \"c90ce63c-e5e9-4edf-8a1c-78ee6d092633\") " pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:31.504081 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-host-slash\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:23:31.504081 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cnibin\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.504081 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503867    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-cni-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.504081 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-registration-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.504081 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysctl-conf\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.504081 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-run\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.504081 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-run-netns\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.504081 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-cnibin\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.504081 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.503981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-socket-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.504081 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-var-lib-kubelet\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.504081 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504053    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovn-node-metrics-cert\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.504615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9bb5657a-bb11-4279-86f6-35118656d990-dbus\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:31.504615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2bbr\" (UniqueName: \"kubernetes.io/projected/7e97eaa5-7df2-484e-a200-95f3fd1114d9-kube-api-access-m2bbr\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.504615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504233    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jrmw\" (UniqueName: \"kubernetes.io/projected/75d5ea8e-8d3d-4113-926c-5b29267d479b-kube-api-access-6jrmw\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.504615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.504615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504318    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-kubelet\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.504615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.504615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.504615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-cni-bin\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.504615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504482    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-multus-certs\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.504615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysctl-d\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.504615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-sys\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.504615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-tuned\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.505113 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.505113 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504699    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-env-overrides\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.505113 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9bb5657a-bb11-4279-86f6-35118656d990-kubelet-config\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:31.505113 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504930    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-log-socket\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.505113 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.505113 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.504983    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-cni-netd\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.505113 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505015    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-system-cni-dir\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.505113 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.505113 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505065    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-system-cni-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.505113 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75d5ea8e-8d3d-4113-926c-5b29267d479b-cni-binary-copy\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.505113 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505112    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-kubelet\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-etc-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-iptables-alerter-script\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxkxw\" (UniqueName: \"kubernetes.io/projected/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-kube-api-access-mxkxw\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505206    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/539550af-7526-447a-b6e3-d268f758aeb1-hosts-file\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505229    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3-agent-certs\") pod \"konnectivity-agent-9t5tl\" (UID: \"4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3\") " pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-slash\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/539550af-7526-447a-b6e3-d268f758aeb1-tmp-dir\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505301    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-hostroot\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-lib-modules\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovnkube-config\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505390    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6da9cb8-1262-4288-a7fb-5991fff5c41a-serviceca\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:23:31.505446 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqsf\" (UniqueName: \"kubernetes.io/projected/a6da9cb8-1262-4288-a7fb-5991fff5c41a-kube-api-access-tfqsf\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:23:31.505893 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:31.505893 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.505491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-var-lib-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.532955 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.532927    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 07:23:31.533125 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.533083    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 07:23:31.533215 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.533204    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 07:23:31.533292 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.533268    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 07:23:31.533354 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.533298    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 07:23:31.533724 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.533705    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7nffm\""
Nov 02 07:23:31.533831 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.533816    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 07:23:31.533906 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.533881    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lptml\""
Nov 02 07:23:31.533988 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.533943    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 07:23:31.534060 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.534037    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 07:23:31.534168 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.534113    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 07:23:31.534225 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.534183    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 07:23:31.534272 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.534248    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 07:23:31.534338 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.534316    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7q84k\""
Nov 02 07:23:31.534429 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.534381    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 07:23:31.534429 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.534420    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 07:23:31.534531 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.534481    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 07:23:31.534646 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.534633    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 07:23:31.534720 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.534706    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tc9dv\""
Nov 02 07:23:31.534804 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.534777    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 07:23:31.534902 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.534885    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfpr4\""
Nov 02 07:23:31.592006 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.591951    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal" event={"ID":"04aaad1dd656465cdf1b219bba0dba7b","Type":"ContainerStarted","Data":"e3d627c6e8421ae277ab8773485b04ecbef13520834f978ec6dcc4191da38a1a"}
Nov 02 07:23:31.593441 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.593418    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal" event={"ID":"0720e6d8914c83a975428ab7b77292e6","Type":"ContainerStarted","Data":"c0403c57e79b789165972e9076d6419b76ed0168408c4779079dc501d6300472"}
Nov 02 07:23:31.606236 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.606236 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-env-overrides\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.606236 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606242    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9bb5657a-bb11-4279-86f6-35118656d990-kubelet-config\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:31.606236 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606270    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-log-socket\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606328    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-log-socket\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606342    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606391    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606396    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9bb5657a-bb11-4279-86f6-35118656d990-kubelet-config\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606412    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-cni-netd\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-system-cni-dir\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606439    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606456    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-system-cni-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606480    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-cni-netd\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75d5ea8e-8d3d-4113-926c-5b29267d479b-cni-binary-copy\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-kubelet\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-etc-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606546    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-system-cni-dir\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-iptables-alerter-script\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:23:31.606575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkxw\" (UniqueName: \"kubernetes.io/projected/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-kube-api-access-mxkxw\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/539550af-7526-447a-b6e3-d268f758aeb1-hosts-file\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3-agent-certs\") pod \"konnectivity-agent-9t5tl\" (UID: \"4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3\") " pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-slash\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/539550af-7526-447a-b6e3-d268f758aeb1-tmp-dir\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-hostroot\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-lib-modules\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovnkube-config\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606723    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-system-cni-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606775    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606815    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-etc-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6da9cb8-1262-4288-a7fb-5991fff5c41a-serviceca\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606980    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-slash\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.607462 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.606981    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-kubelet\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607490    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-hostroot\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607519    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-env-overrides\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607525    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/539550af-7526-447a-b6e3-d268f758aeb1-hosts-file\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqsf\" (UniqueName: \"kubernetes.io/projected/a6da9cb8-1262-4288-a7fb-5991fff5c41a-kube-api-access-tfqsf\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607630    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-lib-modules\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-var-lib-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607725    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zprsb\" (UniqueName: \"kubernetes.io/projected/948eb959-3a16-4ea4-a87b-9497f8bf46d9-kube-api-access-zprsb\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-cni-multus\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-sys-fs\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jc5\" (UniqueName: \"kubernetes.io/projected/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-kube-api-access-v8jc5\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-modprobe-d\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxct4\" (UniqueName: \"kubernetes.io/projected/0059120b-d9ca-462c-a74b-bb99ea44c3d7-kube-api-access-gxct4\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-cni-bin\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607851    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/539550af-7526-447a-b6e3-d268f758aeb1-tmp-dir\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovnkube-script-lib\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.608162 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607890    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-netns\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607913    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3-konnectivity-ca\") pod \"konnectivity-agent-9t5tl\" (UID: \"4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3\") " pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-device-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-node-log\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.607981    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6da9cb8-1262-4288-a7fb-5991fff5c41a-host\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8qvr\" (UniqueName: \"kubernetes.io/projected/539550af-7526-447a-b6e3-d268f758aeb1-kube-api-access-d8qvr\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-conf-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608035    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-daemon-config\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608155    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6da9cb8-1262-4288-a7fb-5991fff5c41a-serviceca\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608180    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-conf-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5bj8\" (UniqueName: \"kubernetes.io/projected/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-kube-api-access-j5bj8\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-systemd-units\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-ovn\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608268    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75d5ea8e-8d3d-4113-926c-5b29267d479b-cni-binary-copy\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608273    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-k8s-cni-cncf-io\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-socket-dir-parent\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.609083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608385    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-etc-kubernetes\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608415    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysconfig\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-host\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-systemd\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608507    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-iptables-alerter-script\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-kubernetes\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-systemd\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608552    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-etc-kubernetes\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608293    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-k8s-cni-cncf-io\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608600    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysconfig\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608611    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-host\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608645    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-daemon-config\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608676    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-cni-bin\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608697    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-socket-dir-parent\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-tmp\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.608727    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjhf\" (UniqueName: \"kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf\") pod \"network-check-target-g88mp\" (UID: \"c90ce63c-e5e9-4edf-8a1c-78ee6d092633\") " pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:31.610037 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-host-slash\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608752    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608759    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.608773    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs podName:0059120b-d9ca-462c-a74b-bb99ea44c3d7 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:32.108758917 +0000 UTC m=+3.077351144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs") pod "network-metrics-daemon-7j98p" (UID: "0059120b-d9ca-462c-a74b-bb99ea44c3d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608787    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-host-slash\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cnibin\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608799    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-systemd\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608809    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cnibin\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.608811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-cni-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-registration-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysctl-conf\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-run\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-run-netns\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-cnibin\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-socket-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609105    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-var-lib-kubelet\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.610937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovn-node-metrics-cert\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609180    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9bb5657a-bb11-4279-86f6-35118656d990-dbus\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2bbr\" (UniqueName: \"kubernetes.io/projected/7e97eaa5-7df2-484e-a200-95f3fd1114d9-kube-api-access-m2bbr\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jrmw\" (UniqueName: \"kubernetes.io/projected/75d5ea8e-8d3d-4113-926c-5b29267d479b-kube-api-access-6jrmw\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609234    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609247    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-cni-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-kubelet\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609276    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-kubelet\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609303    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609024    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6da9cb8-1262-4288-a7fb-5991fff5c41a-host\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609383    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-device-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609420    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-node-log\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609636    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-cni-bin\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-multus-certs\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609676    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovnkube-config\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.611835 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysctl-d\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-sys\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609740    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-tuned\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609746    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-systemd\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.609959    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-var-lib-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610013    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-cni-multus\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610059    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-sys-fs\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610063    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-var-lib-kubelet\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610094    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-run-netns\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610195    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-run\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610203    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-cnibin\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610226    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysctl-conf\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610237    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610286    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9bb5657a-bb11-4279-86f6-35118656d990-dbus\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610294    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610339    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-modprobe-d\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610421    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-socket-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.613618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610423    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-multus-certs\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610458    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-cni-bin\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610530    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-sys\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610557    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-kubernetes\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.610668    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.610717    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret podName:9bb5657a-bb11-4279-86f6-35118656d990 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:32.110700605 +0000 UTC m=+3.079292840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret") pod "global-pull-secret-syncer-rs7kb" (UID: "9bb5657a-bb11-4279-86f6-35118656d990") : object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.610768    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-registration-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.611063    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysctl-d\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.611105    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-ovn\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.611145    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-systemd-units\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.611188    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-netns\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.611793    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3-konnectivity-ca\") pod \"konnectivity-agent-9t5tl\" (UID: \"4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3\") " pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.612033    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovnkube-script-lib\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.612905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-tmp\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.613208    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovn-node-metrics-cert\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.613239    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3-agent-certs\") pod \"konnectivity-agent-9t5tl\" (UID: \"4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3\") " pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:23:31.614204 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.613611    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-tuned\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.636012 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.633906    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkxw\" (UniqueName: \"kubernetes.io/projected/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-kube-api-access-mxkxw\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:23:31.636012 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.635430    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqsf\" (UniqueName: \"kubernetes.io/projected/a6da9cb8-1262-4288-a7fb-5991fff5c41a-kube-api-access-tfqsf\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:23:31.636012 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.635573    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 07:23:31.636012 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.635592    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 07:23:31.636012 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.635606    2576 projected.go:196] Error preparing data for projected volume kube-api-access-crjhf for pod openshift-network-diagnostics/network-check-target-g88mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:31.636012 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:31.635719    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf podName:c90ce63c-e5e9-4edf-8a1c-78ee6d092633 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:32.135697986 +0000 UTC m=+3.104290228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-crjhf" (UniqueName: "kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf") pod "network-check-target-g88mp" (UID: "c90ce63c-e5e9-4edf-8a1c-78ee6d092633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:31.636948 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.636866    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxct4\" (UniqueName: \"kubernetes.io/projected/0059120b-d9ca-462c-a74b-bb99ea44c3d7-kube-api-access-gxct4\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:31.636948 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.636892    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8qvr\" (UniqueName: \"kubernetes.io/projected/539550af-7526-447a-b6e3-d268f758aeb1-kube-api-access-d8qvr\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:23:31.641936 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.641860    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jrmw\" (UniqueName: \"kubernetes.io/projected/75d5ea8e-8d3d-4113-926c-5b29267d479b-kube-api-access-6jrmw\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.642675 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.642652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5bj8\" (UniqueName: \"kubernetes.io/projected/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-kube-api-access-j5bj8\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.642751 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.642699    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jc5\" (UniqueName: \"kubernetes.io/projected/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-kube-api-access-v8jc5\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.643218 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.643185    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2bbr\" (UniqueName: \"kubernetes.io/projected/7e97eaa5-7df2-484e-a200-95f3fd1114d9-kube-api-access-m2bbr\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.644728 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.644707    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zprsb\" (UniqueName: \"kubernetes.io/projected/948eb959-3a16-4ea4-a87b-9497f8bf46d9-kube-api-access-zprsb\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.689083 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.689048    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:23:31.723063 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.723035    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:23:31.728674 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.728645    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:23:31.733877 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:31.733848    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4860b8_d00a_4f6d_a8ac_7be4b73b2ea0.slice/crio-7674727f9ddb2a9032353c555553c9d0c81f4dc11350933d54c9f03f9785637c WatchSource:0}: Error finding container 7674727f9ddb2a9032353c555553c9d0c81f4dc11350933d54c9f03f9785637c: Status 404 returned error can't find the container with id 7674727f9ddb2a9032353c555553c9d0c81f4dc11350933d54c9f03f9785637c
Nov 02 07:23:31.738960 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:31.738934    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a97cd1c_a0a6_4ea6_95de_ffadd05592b2.slice/crio-5aaaf6c2c82047a42dead94444492f089f388cc3f419682550f5cd614c874ac9 WatchSource:0}: Error finding container 5aaaf6c2c82047a42dead94444492f089f388cc3f419682550f5cd614c874ac9: Status 404 returned error can't find the container with id 5aaaf6c2c82047a42dead94444492f089f388cc3f419682550f5cd614c874ac9
Nov 02 07:23:31.741288 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.741265    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:23:31.748625 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.748603    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:23:31.752215 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:31.752194    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6da9cb8_1262_4288_a7fb_5991fff5c41a.slice/crio-cae4447db8674668e0a1383b3e4b2dfdc808bf449b96c59514bea0bb2c9a8ac0 WatchSource:0}: Error finding container cae4447db8674668e0a1383b3e4b2dfdc808bf449b96c59514bea0bb2c9a8ac0: Status 404 returned error can't find the container with id cae4447db8674668e0a1383b3e4b2dfdc808bf449b96c59514bea0bb2c9a8ac0
Nov 02 07:23:31.758043 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.757873    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:23:31.759795 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:31.759751    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d3ac01_f3d4_4c1f_81ee_7ad6f1cc5400.slice/crio-b3ec61b2f9272c2ce53e71912f48ba76d08af83b6fc10424805a98bbef1f46e3 WatchSource:0}: Error finding container b3ec61b2f9272c2ce53e71912f48ba76d08af83b6fc10424805a98bbef1f46e3: Status 404 returned error can't find the container with id b3ec61b2f9272c2ce53e71912f48ba76d08af83b6fc10424805a98bbef1f46e3
Nov 02 07:23:31.768597 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:31.768564    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539550af_7526_447a_b6e3_d268f758aeb1.slice/crio-81925b6b568ae3709ef5db6166c59006ba56e66ad61d7776496b0abaf11e8cbf WatchSource:0}: Error finding container 81925b6b568ae3709ef5db6166c59006ba56e66ad61d7776496b0abaf11e8cbf: Status 404 returned error can't find the container with id 81925b6b568ae3709ef5db6166c59006ba56e66ad61d7776496b0abaf11e8cbf
Nov 02 07:23:31.779460 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.779441    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bp5ct"
Nov 02 07:23:31.781627 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.781600    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:23:31.795151 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.794991    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:23:31.795151 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:31.795013    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e97eaa5_7df2_484e_a200_95f3fd1114d9.slice/crio-7f29cfc958f17f4eab80997a226f20717591bf7c9af2c6bda266dc72af7cda16 WatchSource:0}: Error finding container 7f29cfc958f17f4eab80997a226f20717591bf7c9af2c6bda266dc72af7cda16: Status 404 returned error can't find the container with id 7f29cfc958f17f4eab80997a226f20717591bf7c9af2c6bda266dc72af7cda16
Nov 02 07:23:31.801731 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:31.801710    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:31.805294 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:31.805273    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4741cb0a_8841_40bb_8cbe_5b4cdd7de7d3.slice/crio-5bdcb3c6342c96f5b0fe38bb840d62ce06f3a49a0e7d2e03b2222a964528a197 WatchSource:0}: Error finding container 5bdcb3c6342c96f5b0fe38bb840d62ce06f3a49a0e7d2e03b2222a964528a197: Status 404 returned error can't find the container with id 5bdcb3c6342c96f5b0fe38bb840d62ce06f3a49a0e7d2e03b2222a964528a197
Nov 02 07:23:31.812236 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:31.812211    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948eb959_3a16_4ea4_a87b_9497f8bf46d9.slice/crio-c2b7759057ce265a011dcd82332b2fa9058a5b75714cd040beb4c48ae636e979 WatchSource:0}: Error finding container c2b7759057ce265a011dcd82332b2fa9058a5b75714cd040beb4c48ae636e979: Status 404 returned error can't find the container with id c2b7759057ce265a011dcd82332b2fa9058a5b75714cd040beb4c48ae636e979
Nov 02 07:23:32.113485 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.113451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:32.113677 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.113511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:32.113677 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:32.113632    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:32.113677 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:32.113659    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:32.113843 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:32.113705    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret podName:9bb5657a-bb11-4279-86f6-35118656d990 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:33.113682797 +0000 UTC m=+4.082275044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret") pod "global-pull-secret-syncer-rs7kb" (UID: "9bb5657a-bb11-4279-86f6-35118656d990") : object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:32.113843 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:32.113730    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs podName:0059120b-d9ca-462c-a74b-bb99ea44c3d7 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:33.113718624 +0000 UTC m=+4.082310850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs") pod "network-metrics-daemon-7j98p" (UID: "0059120b-d9ca-462c-a74b-bb99ea44c3d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:32.214184 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.214156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjhf\" (UniqueName: \"kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf\") pod \"network-check-target-g88mp\" (UID: \"c90ce63c-e5e9-4edf-8a1c-78ee6d092633\") " pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:32.214349 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:32.214318    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 07:23:32.214349 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:32.214340    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 07:23:32.214472 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:32.214355    2576 projected.go:196] Error preparing data for projected volume kube-api-access-crjhf for pod openshift-network-diagnostics/network-check-target-g88mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:32.214472 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:32.214408    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf podName:c90ce63c-e5e9-4edf-8a1c-78ee6d092633 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:33.214394675 +0000 UTC m=+4.182986916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-crjhf" (UniqueName: "kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf") pod "network-check-target-g88mp" (UID: "c90ce63c-e5e9-4edf-8a1c-78ee6d092633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:32.482249 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.482156    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 07:18:31 +0000 UTC" deadline="2028-06-25 20:24:26.092000697 +0000 UTC"
Nov 02 07:23:32.482249 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.482202    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14437h0m53.60980405s"
Nov 02 07:23:32.586384 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.585897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:32.586384 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:32.586035    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:32.586384 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.586126    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:32.586384 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:32.586223    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:32.586384 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.586278    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:32.586384 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:32.586352    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:32.615713 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.615679    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5" event={"ID":"948eb959-3a16-4ea4-a87b-9497f8bf46d9","Type":"ContainerStarted","Data":"c2b7759057ce265a011dcd82332b2fa9058a5b75714cd040beb4c48ae636e979"}
Nov 02 07:23:32.618312 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.618267    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k4jxw" event={"ID":"539550af-7526-447a-b6e3-d268f758aeb1","Type":"ContainerStarted","Data":"81925b6b568ae3709ef5db6166c59006ba56e66ad61d7776496b0abaf11e8cbf"}
Nov 02 07:23:32.628146 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.627968    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hd4ds" event={"ID":"a6da9cb8-1262-4288-a7fb-5991fff5c41a","Type":"ContainerStarted","Data":"cae4447db8674668e0a1383b3e4b2dfdc808bf449b96c59514bea0bb2c9a8ac0"}
Nov 02 07:23:32.633644 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.633543    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf" event={"ID":"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0","Type":"ContainerStarted","Data":"7674727f9ddb2a9032353c555553c9d0c81f4dc11350933d54c9f03f9785637c"}
Nov 02 07:23:32.635877 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.635852    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9t5tl" event={"ID":"4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3","Type":"ContainerStarted","Data":"5bdcb3c6342c96f5b0fe38bb840d62ce06f3a49a0e7d2e03b2222a964528a197"}
Nov 02 07:23:32.638206 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.638165    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7x6c" event={"ID":"7e97eaa5-7df2-484e-a200-95f3fd1114d9","Type":"ContainerStarted","Data":"7f29cfc958f17f4eab80997a226f20717591bf7c9af2c6bda266dc72af7cda16"}
Nov 02 07:23:32.641146 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.641000    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bp5ct" event={"ID":"75d5ea8e-8d3d-4113-926c-5b29267d479b","Type":"ContainerStarted","Data":"2d810d41c53333e445450a82ea59d891d3037c2302e589e0e914315efcaae687"}
Nov 02 07:23:32.643410 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.643388    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdfn6" event={"ID":"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400","Type":"ContainerStarted","Data":"b3ec61b2f9272c2ce53e71912f48ba76d08af83b6fc10424805a98bbef1f46e3"}
Nov 02 07:23:32.658355 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:32.658328    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rcfgw" event={"ID":"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2","Type":"ContainerStarted","Data":"5aaaf6c2c82047a42dead94444492f089f388cc3f419682550f5cd614c874ac9"}
Nov 02 07:23:33.131241 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:33.130442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:33.131241 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:33.130501    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:33.131241 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:33.130654    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:33.131241 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:33.130714    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret podName:9bb5657a-bb11-4279-86f6-35118656d990 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:35.130694816 +0000 UTC m=+6.099287067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret") pod "global-pull-secret-syncer-rs7kb" (UID: "9bb5657a-bb11-4279-86f6-35118656d990") : object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:33.131241 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:33.131122    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:33.131241 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:33.131176    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs podName:0059120b-d9ca-462c-a74b-bb99ea44c3d7 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:35.13115806 +0000 UTC m=+6.099750303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs") pod "network-metrics-daemon-7j98p" (UID: "0059120b-d9ca-462c-a74b-bb99ea44c3d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:33.232152 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:33.231372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjhf\" (UniqueName: \"kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf\") pod \"network-check-target-g88mp\" (UID: \"c90ce63c-e5e9-4edf-8a1c-78ee6d092633\") " pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:33.232152 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:33.231550    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 07:23:33.232152 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:33.231574    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 07:23:33.232152 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:33.231655    2576 projected.go:196] Error preparing data for projected volume kube-api-access-crjhf for pod openshift-network-diagnostics/network-check-target-g88mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:33.232152 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:33.231767    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf podName:c90ce63c-e5e9-4edf-8a1c-78ee6d092633 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:35.231746848 +0000 UTC m=+6.200339082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-crjhf" (UniqueName: "kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf") pod "network-check-target-g88mp" (UID: "c90ce63c-e5e9-4edf-8a1c-78ee6d092633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:33.483271 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:33.483094    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 07:18:31 +0000 UTC" deadline="2028-07-10 21:16:05.584184194 +0000 UTC"
Nov 02 07:23:33.483271 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:33.483164    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14797h52m32.101025495s"
Nov 02 07:23:33.683915 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:33.683875    2576 generic.go:356] "Generic (PLEG): container finished" podID="0720e6d8914c83a975428ab7b77292e6" containerID="633489e80e5762c041bcd6e56687d741726ab57e6e84f48b8f4aa66edc5cddff" exitCode=0
Nov 02 07:23:33.684101 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:33.683985    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal" event={"ID":"0720e6d8914c83a975428ab7b77292e6","Type":"ContainerDied","Data":"633489e80e5762c041bcd6e56687d741726ab57e6e84f48b8f4aa66edc5cddff"}
Nov 02 07:23:33.695427 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:33.694927    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal" event={"ID":"04aaad1dd656465cdf1b219bba0dba7b","Type":"ContainerStarted","Data":"57ba4bd339d969f6fcf7f3d7fab3c03699d97ec5af35807ee0fb8009dfe8f810"}
Nov 02 07:23:34.586252 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:34.586175    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:34.586733 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:34.586311    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:34.586733 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:34.586725    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:34.586860 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:34.586836    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:34.587178 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:34.586937    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:34.587178 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:34.587073    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:34.724253 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:34.724218    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal" event={"ID":"0720e6d8914c83a975428ab7b77292e6","Type":"ContainerStarted","Data":"260d77242372869c493530d612d6c580dd4ef651ebf5decc2aba203d40bc3df3"}
Nov 02 07:23:34.767928 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:34.767055    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal" podStartSLOduration=3.76703311 podStartE2EDuration="3.76703311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:33.771266072 +0000 UTC m=+4.739858324" watchObservedRunningTime="2026-11-02 07:23:34.76703311 +0000 UTC m=+5.735625397"
Nov 02 07:23:35.154177 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:35.154090    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:35.154177 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:35.154157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:35.154410 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:35.154292    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:35.154410 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:35.154361    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret podName:9bb5657a-bb11-4279-86f6-35118656d990 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:39.154341715 +0000 UTC m=+10.122933964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret") pod "global-pull-secret-syncer-rs7kb" (UID: "9bb5657a-bb11-4279-86f6-35118656d990") : object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:35.154809 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:35.154788    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:35.154898 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:35.154846    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs podName:0059120b-d9ca-462c-a74b-bb99ea44c3d7 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:39.154829705 +0000 UTC m=+10.123421950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs") pod "network-metrics-daemon-7j98p" (UID: "0059120b-d9ca-462c-a74b-bb99ea44c3d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:35.255406 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:35.255358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjhf\" (UniqueName: \"kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf\") pod \"network-check-target-g88mp\" (UID: \"c90ce63c-e5e9-4edf-8a1c-78ee6d092633\") " pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:35.255605 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:35.255548    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 07:23:35.255605 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:35.255571    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 07:23:35.255605 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:35.255587    2576 projected.go:196] Error preparing data for projected volume kube-api-access-crjhf for pod openshift-network-diagnostics/network-check-target-g88mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:35.255764 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:35.255653    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf podName:c90ce63c-e5e9-4edf-8a1c-78ee6d092633 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:39.255631784 +0000 UTC m=+10.224224031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-crjhf" (UniqueName: "kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf") pod "network-check-target-g88mp" (UID: "c90ce63c-e5e9-4edf-8a1c-78ee6d092633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:35.747255 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:35.747207    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e97eaa5-7df2-484e-a200-95f3fd1114d9" containerID="466e0ea0b929844ae9a6e1926438c828da35daca7977be55390bff579603c201" exitCode=0
Nov 02 07:23:35.748783 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:35.748456    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7x6c" event={"ID":"7e97eaa5-7df2-484e-a200-95f3fd1114d9","Type":"ContainerDied","Data":"466e0ea0b929844ae9a6e1926438c828da35daca7977be55390bff579603c201"}
Nov 02 07:23:35.827977 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:35.827906    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal" podStartSLOduration=4.827884089 podStartE2EDuration="4.827884089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:34.767910017 +0000 UTC m=+5.736502269" watchObservedRunningTime="2026-11-02 07:23:35.827884089 +0000 UTC m=+6.796476341"
Nov 02 07:23:36.586654 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:36.586615    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:36.586837 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:36.586757    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:36.587217 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:36.587196    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:36.587327 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:36.587297    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:36.587386 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:36.587359    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:36.587455 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:36.587434    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:36.758243 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:36.758207    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9t5tl" event={"ID":"4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3","Type":"ContainerStarted","Data":"c453e56bf19eca060fdb11a616af49771adc862f94b48ff0a3ec1cb2c07f84ee"}
Nov 02 07:23:37.763607 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:37.763573    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf" event={"ID":"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0","Type":"ContainerStarted","Data":"f1c7a7d760478de93345fc60304f8800cc6f3fa66fecdc1b6c04b92405483208"}
Nov 02 07:23:37.767667 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:37.767637    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hd4ds" event={"ID":"a6da9cb8-1262-4288-a7fb-5991fff5c41a","Type":"ContainerStarted","Data":"18f1b0c45b9e1cd6348b5888639db4e2092299b7d18fa6797d4bb14e307784c2"}
Nov 02 07:23:37.806347 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:37.806284    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9t5tl" podStartSLOduration=3.7012289750000003 podStartE2EDuration="7.806262847s" totalImagesPullingTime="4.105033872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:36.799203342 +0000 UTC m=+7.767795591" watchObservedRunningTime="2026-11-02 07:23:37.806262847 +0000 UTC m=+8.774855094"
Nov 02 07:23:38.586516 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:38.586444    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:38.586516 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:38.586488    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:38.586823 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:38.586459    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:38.586823 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:38.586570    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:38.586823 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:38.586650    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:38.586823 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:38.586707    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:39.104318 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:39.104279    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 07:23:39.192522 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:39.192442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:39.192682 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:39.192523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:39.192682 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:39.192642    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:39.192795 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:39.192694    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs podName:0059120b-d9ca-462c-a74b-bb99ea44c3d7 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:47.192677791 +0000 UTC m=+18.161270021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs") pod "network-metrics-daemon-7j98p" (UID: "0059120b-d9ca-462c-a74b-bb99ea44c3d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:39.193080 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:39.193060    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:39.193185 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:39.193113    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret podName:9bb5657a-bb11-4279-86f6-35118656d990 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:47.193098687 +0000 UTC m=+18.161690914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret") pod "global-pull-secret-syncer-rs7kb" (UID: "9bb5657a-bb11-4279-86f6-35118656d990") : object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:39.293298 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:39.293246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjhf\" (UniqueName: \"kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf\") pod \"network-check-target-g88mp\" (UID: \"c90ce63c-e5e9-4edf-8a1c-78ee6d092633\") " pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:39.293438 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:39.293399    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 07:23:39.293438 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:39.293417    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 07:23:39.293438 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:39.293430    2576 projected.go:196] Error preparing data for projected volume kube-api-access-crjhf for pod openshift-network-diagnostics/network-check-target-g88mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:39.293556 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:39.293482    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf podName:c90ce63c-e5e9-4edf-8a1c-78ee6d092633 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:47.293464145 +0000 UTC m=+18.262056384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-crjhf" (UniqueName: "kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf") pod "network-check-target-g88mp" (UID: "c90ce63c-e5e9-4edf-8a1c-78ee6d092633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:39.453959 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:39.453802    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T07:23:39.104300054Z","UUID":"2c22f8ec-a6fb-4abd-9c8e-dc12623aadb6","Handler":null,"Name":"","Endpoint":""}
Nov 02 07:23:39.456627 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:39.456367    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 07:23:39.456627 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:39.456389    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 07:23:39.779807 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:39.779727    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf" event={"ID":"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0","Type":"ContainerStarted","Data":"89370dc84dea234ec3dd7f3340960680ac78d75462b3d0dfd82dddba1a690abd"}
Nov 02 07:23:39.782560 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:39.782502    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bp5ct" event={"ID":"75d5ea8e-8d3d-4113-926c-5b29267d479b","Type":"ContainerStarted","Data":"3a676385a54c67d68156b7c831fc6eba55e50656c695d174ea94f575f1d2627e"}
Nov 02 07:23:39.826327 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:39.826249    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hd4ds" podStartSLOduration=4.688023645 podStartE2EDuration="9.826229721s" totalImagesPullingTime="5.138206076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:37.806839837 +0000 UTC m=+8.775432087" watchObservedRunningTime="2026-11-02 07:23:39.826229721 +0000 UTC m=+10.794821966"
Nov 02 07:23:39.826815 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:39.826774    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-bp5ct" podStartSLOduration=2.109861823 podStartE2EDuration="9.826760638s" totalImagesPullingTime="7.716898815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:39.826078702 +0000 UTC m=+10.794670950" watchObservedRunningTime="2026-11-02 07:23:39.826760638 +0000 UTC m=+10.795352886"
Nov 02 07:23:40.585850 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:40.585821    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:40.586178 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:40.585952    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:40.586335 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:40.586318    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:40.586429 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:40.586408    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:40.586479 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:40.586468    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:40.586528 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:40.586511    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:40.787931 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:40.787875    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf" event={"ID":"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0","Type":"ContainerStarted","Data":"25d9903778eb8fe546ac4c1467e7a5218a23c2d5554395e6b9b441717957dd5f"}
Nov 02 07:23:40.793840 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:40.793805    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e97eaa5-7df2-484e-a200-95f3fd1114d9" containerID="210ca17840dd42863fccf558cabeb47fec0e2b714bbb0c156c11b2728455d5f9" exitCode=0
Nov 02 07:23:40.793995 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:40.793881    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7x6c" event={"ID":"7e97eaa5-7df2-484e-a200-95f3fd1114d9","Type":"ContainerDied","Data":"210ca17840dd42863fccf558cabeb47fec0e2b714bbb0c156c11b2728455d5f9"}
Nov 02 07:23:40.797581 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:40.797551    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k4jxw" event={"ID":"539550af-7526-447a-b6e3-d268f758aeb1","Type":"ContainerStarted","Data":"a88bc74f9443408b381e54890722bc607353478510e43cf17d686cf2479f20c7"}
Nov 02 07:23:40.831236 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:40.831158    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf" podStartSLOduration=2.601268829 podStartE2EDuration="10.831145542s" totalImagesPullingTime="8.229876713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:40.830634448 +0000 UTC m=+11.799226697" watchObservedRunningTime="2026-11-02 07:23:40.831145542 +0000 UTC m=+11.799737786"
Nov 02 07:23:40.869039 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:40.868977    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-k4jxw" podStartSLOduration=2.566556126 podStartE2EDuration="10.868960562s" totalImagesPullingTime="8.302404436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:40.86872095 +0000 UTC m=+11.837313201" watchObservedRunningTime="2026-11-02 07:23:40.868960562 +0000 UTC m=+11.837552812"
Nov 02 07:23:41.496323 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:41.496120    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:23:41.496755 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:41.496734    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:23:41.801951 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:41.801119    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdfn6" event={"ID":"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400","Type":"ContainerStarted","Data":"a703cff83e9af5589eb50819816b9a9e4eaca2bd7a6268f1c07bd59533d92b35"}
Nov 02 07:23:42.586342 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:42.586303    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:42.586505 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:42.586314    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:42.586505 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:42.586420    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:42.586505 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:42.586322    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:42.586505 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:42.586480    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:42.586662 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:42.586593    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:42.805734 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:42.805693    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e97eaa5-7df2-484e-a200-95f3fd1114d9" containerID="8f8c664f78293cee4b5912f39133771d412e94507d894b5789d20ce92ba0b822" exitCode=0
Nov 02 07:23:42.806378 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:42.805789    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7x6c" event={"ID":"7e97eaa5-7df2-484e-a200-95f3fd1114d9","Type":"ContainerDied","Data":"8f8c664f78293cee4b5912f39133771d412e94507d894b5789d20ce92ba0b822"}
Nov 02 07:23:42.807688 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:42.807660    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rcfgw" event={"ID":"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2","Type":"ContainerStarted","Data":"7354779ca3a0f4a87f791b77647a56af782151b30d833f7943c2b542ba22d61b"}
Nov 02 07:23:42.876597 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:42.876497    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zdfn6" podStartSLOduration=4.801666242 podStartE2EDuration="12.876477888s" totalImagesPullingTime="8.074811646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:41.839488433 +0000 UTC m=+12.808080680" watchObservedRunningTime="2026-11-02 07:23:42.876477888 +0000 UTC m=+13.845070139"
Nov 02 07:23:42.905864 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:42.905803    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rcfgw" podStartSLOduration=2.456498537 podStartE2EDuration="12.905784979s" totalImagesPullingTime="10.449286442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:42.905218743 +0000 UTC m=+13.873810989" watchObservedRunningTime="2026-11-02 07:23:42.905784979 +0000 UTC m=+13.874377229"
Nov 02 07:23:43.812640 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:43.812561    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e97eaa5-7df2-484e-a200-95f3fd1114d9" containerID="564e7b047e7de445095c5149bc08b3845876effa8acb368d3ffe86e5c291f936" exitCode=0
Nov 02 07:23:43.812640 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:43.812597    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7x6c" event={"ID":"7e97eaa5-7df2-484e-a200-95f3fd1114d9","Type":"ContainerDied","Data":"564e7b047e7de445095c5149bc08b3845876effa8acb368d3ffe86e5c291f936"}
Nov 02 07:23:44.586441 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:44.586407    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:44.586615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:44.586469    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:44.586615 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:44.586584    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:44.586615 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:44.586598    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:44.586762 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:44.586653    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:44.586799 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:44.586752    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:45.761487 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:45.761297    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:23:45.761854 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:45.761637    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 07:23:45.762226 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:45.762206    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:23:45.820642 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:45.820525    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e97eaa5-7df2-484e-a200-95f3fd1114d9" containerID="d5b606a356bce51db58ab5de1375da4944a6410986ac9cb381684414a1bee06e" exitCode=0
Nov 02 07:23:45.820642 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:45.820613    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7x6c" event={"ID":"7e97eaa5-7df2-484e-a200-95f3fd1114d9","Type":"ContainerDied","Data":"d5b606a356bce51db58ab5de1375da4944a6410986ac9cb381684414a1bee06e"}
Nov 02 07:23:46.586298 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:46.586276    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:46.586382 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:46.586310    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:46.586382 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:46.586374    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:46.586476 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:46.586426    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:46.586564 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:46.586542    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:46.586656 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:46.586630    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:46.826038 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:46.825842    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5" event={"ID":"948eb959-3a16-4ea4-a87b-9497f8bf46d9","Type":"ContainerStarted","Data":"062591b98d14b91bed6082c3d0430d5e9b659e21e8c262d0f8f97995421accc5"}
Nov 02 07:23:46.826415 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:46.826078    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5" event={"ID":"948eb959-3a16-4ea4-a87b-9497f8bf46d9","Type":"ContainerStarted","Data":"e3ca9389ba2b521e5bec13cec1751f48db1d88aaea4969c764ee08bb61ea4200"}
Nov 02 07:23:46.826415 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:46.826116    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5" event={"ID":"948eb959-3a16-4ea4-a87b-9497f8bf46d9","Type":"ContainerStarted","Data":"4a149c4b3ed33db29cafe83e21fe7c2b2a09612f56057a6c0c1a3456fac212f1"}
Nov 02 07:23:46.826415 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:46.826154    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5" event={"ID":"948eb959-3a16-4ea4-a87b-9497f8bf46d9","Type":"ContainerStarted","Data":"a76b8a8ad296ac5ca2181b668f2720892e67c48e07445c8e0d3cfd9b4b1ae8a1"}
Nov 02 07:23:46.826415 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:46.826190    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5" event={"ID":"948eb959-3a16-4ea4-a87b-9497f8bf46d9","Type":"ContainerStarted","Data":"ef0217b0279f7e7c1510bef104bf266e004b2e48df20777243bc862916964ae2"}
Nov 02 07:23:46.826415 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:46.826231    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5" event={"ID":"948eb959-3a16-4ea4-a87b-9497f8bf46d9","Type":"ContainerStarted","Data":"2a8222c0c68a1d95d3517cee8acd66c677ac9b7c65445b01240220a0d21d8991"}
Nov 02 07:23:46.829291 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:46.829260    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e97eaa5-7df2-484e-a200-95f3fd1114d9" containerID="64ff9ebf7a65dbf37e3f3652e2a63d713722bdbd23f9465173df0f51b07e46b3" exitCode=0
Nov 02 07:23:46.829291 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:46.829289    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7x6c" event={"ID":"7e97eaa5-7df2-484e-a200-95f3fd1114d9","Type":"ContainerDied","Data":"64ff9ebf7a65dbf37e3f3652e2a63d713722bdbd23f9465173df0f51b07e46b3"}
Nov 02 07:23:47.252865 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:47.252827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:47.253038 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:47.252875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:47.253038 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:47.252982    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:47.253038 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:47.252985    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:47.253038 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:47.253036    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret podName:9bb5657a-bb11-4279-86f6-35118656d990 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:03.253019564 +0000 UTC m=+34.221611804 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret") pod "global-pull-secret-syncer-rs7kb" (UID: "9bb5657a-bb11-4279-86f6-35118656d990") : object "kube-system"/"original-pull-secret" not registered
Nov 02 07:23:47.253191 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:47.253052    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs podName:0059120b-d9ca-462c-a74b-bb99ea44c3d7 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:03.253044635 +0000 UTC m=+34.221636860 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs") pod "network-metrics-daemon-7j98p" (UID: "0059120b-d9ca-462c-a74b-bb99ea44c3d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 07:23:47.354030 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:47.353999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjhf\" (UniqueName: \"kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf\") pod \"network-check-target-g88mp\" (UID: \"c90ce63c-e5e9-4edf-8a1c-78ee6d092633\") " pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:47.354152 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:47.354110    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 07:23:47.354152 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:47.354123    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 07:23:47.354152 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:47.354132    2576 projected.go:196] Error preparing data for projected volume kube-api-access-crjhf for pod openshift-network-diagnostics/network-check-target-g88mp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:47.354249 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:47.354174    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf podName:c90ce63c-e5e9-4edf-8a1c-78ee6d092633 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:03.354162605 +0000 UTC m=+34.322754843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-crjhf" (UniqueName: "kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf") pod "network-check-target-g88mp" (UID: "c90ce63c-e5e9-4edf-8a1c-78ee6d092633") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 07:23:47.834591 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:47.834559    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7x6c" event={"ID":"7e97eaa5-7df2-484e-a200-95f3fd1114d9","Type":"ContainerStarted","Data":"6aa4efe125c0466332ca47d81f75862be93c3adf295469c03374b816ccdbc682"}
Nov 02 07:23:47.904207 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:47.904168    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g7x6c" podStartSLOduration=7.285176653 podStartE2EDuration="17.904156665s" totalImagesPullingTime="10.618980012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:47.903970891 +0000 UTC m=+18.872563139" watchObservedRunningTime="2026-11-02 07:23:47.904156665 +0000 UTC m=+18.872748911"
Nov 02 07:23:48.585800 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:48.585773    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:48.585800 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:48.585803    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:48.586003 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:48.585773    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:48.586003 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:48.585858    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:48.586003 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:48.585972    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:48.586086 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:48.586042    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:49.843658 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:49.843620    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5" event={"ID":"948eb959-3a16-4ea4-a87b-9497f8bf46d9","Type":"ContainerStarted","Data":"ddd7285b0363e1d333563227551f063df4f7277de65cacaf72a877c070c7c557"}
Nov 02 07:23:50.586169 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:50.586139    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:50.586316 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:50.586234    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:50.586316 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:50.586246    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:50.586316 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:50.586305    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:50.586450 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:50.586343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:50.586450 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:50.586429    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:51.850657 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:51.850455    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5" event={"ID":"948eb959-3a16-4ea4-a87b-9497f8bf46d9","Type":"ContainerStarted","Data":"09909a14a0253dd4875d41b5f078bfa71b81890f3c18c7f293067bac67672291"}
Nov 02 07:23:51.851372 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:51.850709    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:51.851372 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:51.850747    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:51.851372 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:51.850780    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:51.866285 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:51.866261    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:51.866403 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:51.866362    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:23:51.939338 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:51.939286    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5" podStartSLOduration=7.284255612 podStartE2EDuration="21.939274163s" totalImagesPullingTime="14.655018551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:51.938731558 +0000 UTC m=+22.907323804" watchObservedRunningTime="2026-11-02 07:23:51.939274163 +0000 UTC m=+22.907866436"
Nov 02 07:23:52.585647 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:52.585608    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:52.585810 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:52.585653    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:52.585810 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:52.585653    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:52.585810 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:52.585728    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:52.585975 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:52.585809    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:52.585975 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:52.585862    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:52.988829 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:52.988792    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rs7kb"]
Nov 02 07:23:52.989388 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:52.989024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:52.989388 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:52.989158    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:52.991714 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:52.991673    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g88mp"]
Nov 02 07:23:52.991823 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:52.991803    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:52.991959 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:52.991938    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:52.994496 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:52.994472    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7j98p"]
Nov 02 07:23:52.994591 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:52.994562    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:52.994647 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:52.994630    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:54.586036 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:54.585994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:54.586496 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:54.585994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:54.586496 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:54.586110    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:54.586496 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:54.586202    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:54.586496 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:54.585994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:54.586496 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:54.586297    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:56.585858 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:56.585825    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:56.586532 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:56.585947    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:56.586532 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:56.585965    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g88mp" podUID="c90ce63c-e5e9-4edf-8a1c-78ee6d092633"
Nov 02 07:23:56.586532 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:56.585984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:56.586532 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:56.586059    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rs7kb" podUID="9bb5657a-bb11-4279-86f6-35118656d990"
Nov 02 07:23:56.586532 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:56.586123    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7j98p" podUID="0059120b-d9ca-462c-a74b-bb99ea44c3d7"
Nov 02 07:23:58.384516 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.384486    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeReady"
Nov 02 07:23:58.384917 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.384588    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 07:23:58.431236 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.431194    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zjjxl"]
Nov 02 07:23:58.434461 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.434442    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w6xgl"]
Nov 02 07:23:58.434596 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.434578    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zjjxl"
Nov 02 07:23:58.437261 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.437244    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nfdng"]
Nov 02 07:23:58.437406 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.437385    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w6xgl"
Nov 02 07:23:58.439927 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.439913    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-649bdd4b4b-st5hm"]
Nov 02 07:23:58.440035 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.440021    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nfdng"
Nov 02 07:23:58.442642 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.442623    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc"]
Nov 02 07:23:58.442849 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.442834    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.445813 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.445796    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"]
Nov 02 07:23:58.445943 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.445926    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc"
Nov 02 07:23:58.449645 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.449626    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-769444d69f-x9wg9"]
Nov 02 07:23:58.449766 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.449693    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.452543 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.452528    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"]
Nov 02 07:23:58.452672 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.452655    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.454819 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.454806    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 07:23:58.455142 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.455130    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"]
Nov 02 07:23:58.455252 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.455237    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:23:58.455950 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.455935    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 07:23:58.459222 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.459197    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6dd94bff97-89klj"]
Nov 02 07:23:58.463993 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.463971    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 07:23:58.464320 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.464302    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v4hqr\""
Nov 02 07:23:58.464902 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.464870    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 07:23:58.465115 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.465088    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 07:23:58.466842 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.466747    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tct4z"]
Nov 02 07:23:58.466842 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.466779    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:23:58.469491 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.469470    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.476356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.475637    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"]
Nov 02 07:23:58.476356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.475877    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.480133 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.480071    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"]
Nov 02 07:23:58.480395 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.480372    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:23:58.486040 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.486022    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"]
Nov 02 07:23:58.486387 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.486198    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:23:58.489150 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.489133    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc"]
Nov 02 07:23:58.489273 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.489258    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:23:58.493942 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.493921    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5tv7w"]
Nov 02 07:23:58.494128 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.494111    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc"
Nov 02 07:23:58.497119 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.497101    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-frv99"]
Nov 02 07:23:58.497320 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.497292    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:58.505517 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505494    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w6xgl"]
Nov 02 07:23:58.505618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505533    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zjjxl"]
Nov 02 07:23:58.505618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505556    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nfdng"]
Nov 02 07:23:58.505618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505576    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-649bdd4b4b-st5hm"]
Nov 02 07:23:58.505618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505595    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"]
Nov 02 07:23:58.505618 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505614    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"]
Nov 02 07:23:58.507191 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505633    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tct4z"]
Nov 02 07:23:58.507191 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505675    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"]
Nov 02 07:23:58.507191 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505701    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6dd94bff97-89klj"]
Nov 02 07:23:58.507191 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505724    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"]
Nov 02 07:23:58.507191 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505743    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc"]
Nov 02 07:23:58.507191 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505762    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"]
Nov 02 07:23:58.507191 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505782    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-769444d69f-x9wg9"]
Nov 02 07:23:58.507191 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505802    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc"]
Nov 02 07:23:58.507191 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505820    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-frv99"]
Nov 02 07:23:58.507191 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.505918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-frv99"
Nov 02 07:23:58.508217 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.508197    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5tv7w"]
Nov 02 07:23:58.508810 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.508790    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"]
Nov 02 07:23:58.533359 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.533341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.533456 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.533365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:23:58.533456 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.533385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhmb\" (UniqueName: \"kubernetes.io/projected/437c4525-b041-4931-ba40-99c518193657-kube-api-access-ldhmb\") pod \"kas-connection-checker-685895c7c9-w6xgl\" (UID: \"437c4525-b041-4931-ba40-99c518193657\") " pod="kube-system/kas-connection-checker-685895c7c9-w6xgl"
Nov 02 07:23:58.533456 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.533402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f4ba5ec9-5277-460a-976e-3b1f154f7623-config\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.533594 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.533467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ff6ca66-e0d8-441f-8822-022929c12120-ca-trust-extracted\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.533594 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.533502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-registry-certificates\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.533594 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.533570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/06de30c9-de01-4f63-859a-23df97972617-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:23:58.533730 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.533607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-bound-sa-token\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.533730 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.533691    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cqp8\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-kube-api-access-9cqp8\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.533795 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.533771    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqpcg\" (UniqueName: \"kubernetes.io/projected/b93e0f22-01e7-4128-a999-da9b96e029f8-kube-api-access-qqpcg\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.533849 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.533834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbdl7\" (UniqueName: \"kubernetes.io/projected/f4ba5ec9-5277-460a-976e-3b1f154f7623-kube-api-access-dbdl7\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.533933 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.533912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-bound-sa-token\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.534017 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg6jk\" (UniqueName: \"kubernetes.io/projected/78f8157e-9211-4f25-886e-4a9e9338e730-kube-api-access-mg6jk\") pod \"kas-connection-checker-685895c7c9-zjjxl\" (UID: \"78f8157e-9211-4f25-886e-4a9e9338e730\") " pod="kube-system/kas-connection-checker-685895c7c9-zjjxl"
Nov 02 07:23:58.534073 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.534139 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/124a32e6-1529-4b84-aa39-dda6f9332364-ca-trust-extracted\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.534224 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fwtm\" (UniqueName: \"kubernetes.io/projected/d338565e-a9f0-44e8-ba6c-e0d88c253a10-kube-api-access-7fwtm\") pod \"volume-data-source-validator-58c9f5f46d-b2tsc\" (UID: \"d338565e-a9f0-44e8-ba6c-e0d88c253a10\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc"
Nov 02 07:23:58.534281 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0196d04c-bd2b-48c7-bdb9-d31d83753247-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:23:58.534342 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-image-registry-private-configuration\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.534342 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdn4r\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-kube-api-access-rdn4r\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.534416 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-trusted-ca\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.534458 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4ba5ec9-5277-460a-976e-3b1f154f7623-serving-cert\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.534490 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-stats-auth\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.534531 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-registry-certificates\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.534564 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-default-certificate\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.534602 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.534642 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.534679 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-image-registry-private-configuration\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.534679 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-trusted-ca\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.534745 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-installation-pull-secrets\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.534745 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-installation-pull-secrets\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.534813 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534756    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0196d04c-bd2b-48c7-bdb9-d31d83753247-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:23:58.534813 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534770    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4pcq\" (UniqueName: \"kubernetes.io/projected/0196d04c-bd2b-48c7-bdb9-d31d83753247-kube-api-access-d4pcq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:23:58.534813 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534785    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqvfj\" (UniqueName: \"kubernetes.io/projected/f85f18d7-a8b8-4634-af97-eb4e71dc5000-kube-api-access-rqvfj\") pod \"kas-connection-checker-685895c7c9-nfdng\" (UID: \"f85f18d7-a8b8-4634-af97-eb4e71dc5000\") " pod="kube-system/kas-connection-checker-685895c7c9-nfdng"
Nov 02 07:23:58.534937 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.534864    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4ba5ec9-5277-460a-976e-3b1f154f7623-trusted-ca\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.555428 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.555410    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 07:23:58.555530 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.555450    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 07:23:58.555614 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.555591    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 07:23:58.555678 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.555645    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:23:58.555678 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.555660    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 07:23:58.555678 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.555662    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 07:23:58.556220 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.556193    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 07:23:58.556319 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.556221    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 07:23:58.556319 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.556237    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:23:58.556562 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.556547    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 07:23:58.556692 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.556675    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 07:23:58.556934 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.556918    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 07:23:58.557496 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.557479    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:23:58.557688 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.557671    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 07:23:58.557757 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.557720    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:23:58.558029 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.558011    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 07:23:58.558228 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.558210    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 07:23:58.558298 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.558273    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 07:23:58.558647 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.558630    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 07:23:58.558784 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.558770    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 07:23:58.558855 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.558777    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 07:23:58.559012 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.558996    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 07:23:58.559119 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.559098    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 07:23:58.559302 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.559283    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 07:23:58.559302 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.559294    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 07:23:58.559430 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.559307    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 07:23:58.559430 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.559334    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:23:58.559430 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.559371    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 07:23:58.580629 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.580610    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 07:23:58.581116 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581103    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 07:23:58.581235 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581219    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 07:23:58.581273 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581245    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 07:23:58.581273 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581252    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 07:23:58.581359 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581328    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 07:23:58.581359 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581345    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 07:23:58.581485 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581470    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 07:23:58.581534 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581487    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 07:23:58.581534 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581505    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 07:23:58.581534 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581511    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 07:23:58.581534 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581519    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 07:23:58.581724 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581706    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 07:23:58.581910 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581760    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7x2d2\""
Nov 02 07:23:58.581910 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581766    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lsjmf\""
Nov 02 07:23:58.581910 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581762    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zwv54\""
Nov 02 07:23:58.581910 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.581871    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jcjcp\""
Nov 02 07:23:58.582139 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.582019    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2fd7x\""
Nov 02 07:23:58.582139 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.582085    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-6z8kh\""
Nov 02 07:23:58.582139 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.582113    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7q455\""
Nov 02 07:23:58.582139 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.582121    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tj7f2\""
Nov 02 07:23:58.582139 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.582128    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-v8lmk\""
Nov 02 07:23:58.582329 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.582150    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ztvhn\""
Nov 02 07:23:58.582329 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.582130    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zjjlg\""
Nov 02 07:23:58.582329 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.582128    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-grjcl\""
Nov 02 07:23:58.582455 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.582346    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 07:23:58.582455 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.582403    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 07:23:58.582455 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.582443    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 07:23:58.586488 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.586473    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:23:58.586536 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.586499    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:23:58.586683 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.586669    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:23:58.635760 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.635713    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl4ww\" (UniqueName: \"kubernetes.io/projected/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-kube-api-access-jl4ww\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.635760 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.635744    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg6jk\" (UniqueName: \"kubernetes.io/projected/78f8157e-9211-4f25-886e-4a9e9338e730-kube-api-access-mg6jk\") pod \"kas-connection-checker-685895c7c9-zjjxl\" (UID: \"78f8157e-9211-4f25-886e-4a9e9338e730\") " pod="kube-system/kas-connection-checker-685895c7c9-zjjxl"
Nov 02 07:23:58.635870 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.635767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.635870 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.635786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sd8w\" (UniqueName: \"kubernetes.io/projected/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-kube-api-access-9sd8w\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:23:58.635870 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.635857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/124a32e6-1529-4b84-aa39-dda6f9332364-ca-trust-extracted\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.635984 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.635927    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 07:23:58.635984 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.635968    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:59.13595482 +0000 UTC m=+30.104547078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : secret "router-metrics-certs-default" not found
Nov 02 07:23:58.636125 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fwtm\" (UniqueName: \"kubernetes.io/projected/d338565e-a9f0-44e8-ba6c-e0d88c253a10-kube-api-access-7fwtm\") pod \"volume-data-source-validator-58c9f5f46d-b2tsc\" (UID: \"d338565e-a9f0-44e8-ba6c-e0d88c253a10\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc"
Nov 02 07:23:58.636125 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0196d04c-bd2b-48c7-bdb9-d31d83753247-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:23:58.636125 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-image-registry-private-configuration\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.636125 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdn4r\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-kube-api-access-rdn4r\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.636125 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2qj8\" (UniqueName: \"kubernetes.io/projected/54a48a78-5f18-4e96-be7b-36f356cc9b39-kube-api-access-k2qj8\") pod \"ingress-canary-frv99\" (UID: \"54a48a78-5f18-4e96-be7b-36f356cc9b39\") " pod="openshift-ingress-canary/ingress-canary-frv99"
Nov 02 07:23:58.636361 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636208    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/124a32e6-1529-4b84-aa39-dda6f9332364-ca-trust-extracted\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.636361 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-service-ca-bundle\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.636361 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx642\" (UniqueName: \"kubernetes.io/projected/7bf60630-1d98-4928-a2bf-dda7a1e1a110-kube-api-access-rx642\") pod \"network-check-source-6f78d59f99-lrtvc\" (UID: \"7bf60630-1d98-4928-a2bf-dda7a1e1a110\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc"
Nov 02 07:23:58.636361 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636307    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-trusted-ca\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.636361 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4ba5ec9-5277-460a-976e-3b1f154f7623-serving-cert\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.636592 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-stats-auth\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.636592 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-registry-certificates\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.636592 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-default-certificate\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.636592 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.636592 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-serving-cert\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.636592 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.636592 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.636592 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-tmp-dir\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:58.636592 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636590    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-image-registry-private-configuration\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636616    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-trusted-ca\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636646    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0196d04c-bd2b-48c7-bdb9-d31d83753247-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.636662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.636692    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.636706    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769444d69f-x9wg9: secret "image-registry-tls" not found
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.636777    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls podName:9ff6ca66-e0d8-441f-8822-022929c12120 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:59.136761417 +0000 UTC m=+30.105353667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls") pod "image-registry-769444d69f-x9wg9" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120") : secret "image-registry-tls" not found
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.636855    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.636869    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649bdd4b4b-st5hm: secret "image-registry-tls" not found
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.636925    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls podName:124a32e6-1529-4b84-aa39-dda6f9332364 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:59.136911858 +0000 UTC m=+30.105504097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls") pod "image-registry-649bdd4b4b-st5hm" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364") : secret "image-registry-tls" not found
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637204    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-registry-certificates\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637291    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh46m\" (UniqueName: \"kubernetes.io/projected/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-kube-api-access-qh46m\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-installation-pull-secrets\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-installation-pull-secrets\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637495    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0196d04c-bd2b-48c7-bdb9-d31d83753247-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:23:58.637558 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637517    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4pcq\" (UniqueName: \"kubernetes.io/projected/0196d04c-bd2b-48c7-bdb9-d31d83753247-kube-api-access-d4pcq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvfj\" (UniqueName: \"kubernetes.io/projected/f85f18d7-a8b8-4634-af97-eb4e71dc5000-kube-api-access-rqvfj\") pod \"kas-connection-checker-685895c7c9-nfdng\" (UID: \"f85f18d7-a8b8-4634-af97-eb4e71dc5000\") " pod="kube-system/kas-connection-checker-685895c7c9-nfdng"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4ba5ec9-5277-460a-976e-3b1f154f7623-trusted-ca\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637598    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm5vh\" (UniqueName: \"kubernetes.io/projected/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-kube-api-access-bm5vh\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637605    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-trusted-ca\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/531c943e-bc09-4336-9d48-065b8b716dc4-config\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-snapshots\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbfdt\" (UniqueName: \"kubernetes.io/projected/531c943e-bc09-4336-9d48-065b8b716dc4-kube-api-access-vbfdt\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637742    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhmb\" (UniqueName: \"kubernetes.io/projected/437c4525-b041-4931-ba40-99c518193657-kube-api-access-ldhmb\") pod \"kas-connection-checker-685895c7c9-w6xgl\" (UID: \"437c4525-b041-4931-ba40-99c518193657\") " pod="kube-system/kas-connection-checker-685895c7c9-w6xgl"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f4ba5ec9-5277-460a-976e-3b1f154f7623-config\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637882    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/531c943e-bc09-4336-9d48-065b8b716dc4-serving-cert\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637913    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-trusted-ca\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637924    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-tmp\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.638356 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ff6ca66-e0d8-441f-8822-022929c12120-ca-trust-extracted\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-registry-certificates\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.637991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.638002    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.638013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-config-volume\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.638040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/06de30c9-de01-4f63-859a-23df97972617-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.638057    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert podName:06de30c9-de01-4f63-859a-23df97972617 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:59.138041569 +0000 UTC m=+30.106633800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wlj5v" (UID: "06de30c9-de01-4f63-859a-23df97972617") : secret "networking-console-plugin-cert" not found
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.638088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-bound-sa-token\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.638121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cqp8\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-kube-api-access-9cqp8\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.638149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqpcg\" (UniqueName: \"kubernetes.io/projected/b93e0f22-01e7-4128-a999-da9b96e029f8-kube-api-access-qqpcg\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.638176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54a48a78-5f18-4e96-be7b-36f356cc9b39-cert\") pod \"ingress-canary-frv99\" (UID: \"54a48a78-5f18-4e96-be7b-36f356cc9b39\") " pod="openshift-ingress-canary/ingress-canary-frv99"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.638202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.638248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbdl7\" (UniqueName: \"kubernetes.io/projected/f4ba5ec9-5277-460a-976e-3b1f154f7623-kube-api-access-dbdl7\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.638273    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-bound-sa-token\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.638655    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/06de30c9-de01-4f63-859a-23df97972617-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:23:58.639138 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.638828    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4ba5ec9-5277-460a-976e-3b1f154f7623-trusted-ca\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.639856 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.639260    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ff6ca66-e0d8-441f-8822-022929c12120-ca-trust-extracted\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.639856 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.639427    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f4ba5ec9-5277-460a-976e-3b1f154f7623-config\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.639856 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.639684    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-registry-certificates\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.639856 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.639793    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:23:59.139779108 +0000 UTC m=+30.108371354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : configmap references non-existent config key: service-ca.crt
Nov 02 07:23:58.641263 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.641199    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-image-registry-private-configuration\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.641263 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.641215    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-stats-auth\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.641263 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.641215    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-default-certificate\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.641263 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.641216    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4ba5ec9-5277-460a-976e-3b1f154f7623-serving-cert\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.641661 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.641642    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-installation-pull-secrets\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.641980 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.641961    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-installation-pull-secrets\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.642046 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.642013    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-image-registry-private-configuration\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.642576 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.642552    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0196d04c-bd2b-48c7-bdb9-d31d83753247-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:23:58.660525 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.660503    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 07:23:58.660656 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.660593    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zwm44\""
Nov 02 07:23:58.660656 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.660637    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhg4q\""
Nov 02 07:23:58.660850 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.660805    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hdb4w\""
Nov 02 07:23:58.660946 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.660933    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 07:23:58.675015 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.674972    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvfj\" (UniqueName: \"kubernetes.io/projected/f85f18d7-a8b8-4634-af97-eb4e71dc5000-kube-api-access-rqvfj\") pod \"kas-connection-checker-685895c7c9-nfdng\" (UID: \"f85f18d7-a8b8-4634-af97-eb4e71dc5000\") " pod="kube-system/kas-connection-checker-685895c7c9-nfdng"
Nov 02 07:23:58.675430 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.675404    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg6jk\" (UniqueName: \"kubernetes.io/projected/78f8157e-9211-4f25-886e-4a9e9338e730-kube-api-access-mg6jk\") pod \"kas-connection-checker-685895c7c9-zjjxl\" (UID: \"78f8157e-9211-4f25-886e-4a9e9338e730\") " pod="kube-system/kas-connection-checker-685895c7c9-zjjxl"
Nov 02 07:23:58.675536 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.675521    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4pcq\" (UniqueName: \"kubernetes.io/projected/0196d04c-bd2b-48c7-bdb9-d31d83753247-kube-api-access-d4pcq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:23:58.675581 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.675563    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdn4r\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-kube-api-access-rdn4r\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.676648 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.676625    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cqp8\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-kube-api-access-9cqp8\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.678581 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.678560    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-bound-sa-token\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:58.678680 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.678654    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbdl7\" (UniqueName: \"kubernetes.io/projected/f4ba5ec9-5277-460a-976e-3b1f154f7623-kube-api-access-dbdl7\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.678740 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.678569    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqpcg\" (UniqueName: \"kubernetes.io/projected/b93e0f22-01e7-4128-a999-da9b96e029f8-kube-api-access-qqpcg\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:58.679005 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.678989    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-bound-sa-token\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:58.680217 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.680195    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhmb\" (UniqueName: \"kubernetes.io/projected/437c4525-b041-4931-ba40-99c518193657-kube-api-access-ldhmb\") pod \"kas-connection-checker-685895c7c9-w6xgl\" (UID: \"437c4525-b041-4931-ba40-99c518193657\") " pod="kube-system/kas-connection-checker-685895c7c9-w6xgl"
Nov 02 07:23:58.680527 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.680505    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fwtm\" (UniqueName: \"kubernetes.io/projected/d338565e-a9f0-44e8-ba6c-e0d88c253a10-kube-api-access-7fwtm\") pod \"volume-data-source-validator-58c9f5f46d-b2tsc\" (UID: \"d338565e-a9f0-44e8-ba6c-e0d88c253a10\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc"
Nov 02 07:23:58.739080 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-service-ca-bundle\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.739193 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx642\" (UniqueName: \"kubernetes.io/projected/7bf60630-1d98-4928-a2bf-dda7a1e1a110-kube-api-access-rx642\") pod \"network-check-source-6f78d59f99-lrtvc\" (UID: \"7bf60630-1d98-4928-a2bf-dda7a1e1a110\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc"
Nov 02 07:23:58.739193 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.739193 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-serving-cert\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.739333 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739315    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-tmp-dir\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:58.739478 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:23:58.739557 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739501    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:58.739624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh46m\" (UniqueName: \"kubernetes.io/projected/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-kube-api-access-qh46m\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:23:58.739624 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739614    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-tmp-dir\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:58.739725 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739636    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm5vh\" (UniqueName: \"kubernetes.io/projected/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-kube-api-access-bm5vh\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:58.739725 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/531c943e-bc09-4336-9d48-065b8b716dc4-config\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:23:58.739725 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739684    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-service-ca-bundle\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.739725 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.739694    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:23:58.739725 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-snapshots\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.739981 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbfdt\" (UniqueName: \"kubernetes.io/projected/531c943e-bc09-4336-9d48-065b8b716dc4-kube-api-access-vbfdt\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:23:58.739981 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.739753    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls podName:08bc1af6-1c3b-4740-9dc0-fa309e83b78a nodeName:}" failed. No retries permitted until 2026-11-02 07:23:59.239737501 +0000 UTC m=+30.208329731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls") pod "dns-default-5tv7w" (UID: "08bc1af6-1c3b-4740-9dc0-fa309e83b78a") : secret "dns-default-metrics-tls" not found
Nov 02 07:23:58.739981 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:23:58.739981 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739833    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/531c943e-bc09-4336-9d48-065b8b716dc4-serving-cert\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:23:58.739981 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-tmp\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.739981 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:23:58.739981 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-config-volume\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:58.739981 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54a48a78-5f18-4e96-be7b-36f356cc9b39-cert\") pod \"ingress-canary-frv99\" (UID: \"54a48a78-5f18-4e96-be7b-36f356cc9b39\") " pod="openshift-ingress-canary/ingress-canary-frv99"
Nov 02 07:23:58.739981 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.739976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:23:58.740419 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.740015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl4ww\" (UniqueName: \"kubernetes.io/projected/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-kube-api-access-jl4ww\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.740419 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.740057    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sd8w\" (UniqueName: \"kubernetes.io/projected/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-kube-api-access-9sd8w\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:23:58.740419 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.740066    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.740419 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.740132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qj8\" (UniqueName: \"kubernetes.io/projected/54a48a78-5f18-4e96-be7b-36f356cc9b39-kube-api-access-k2qj8\") pod \"ingress-canary-frv99\" (UID: \"54a48a78-5f18-4e96-be7b-36f356cc9b39\") " pod="openshift-ingress-canary/ingress-canary-frv99"
Nov 02 07:23:58.740419 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.740235    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:23:58.740419 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.740265    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-snapshots\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.740419 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.740359    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/531c943e-bc09-4336-9d48-065b8b716dc4-config\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:23:58.740419 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.740359    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 07:23:58.740801 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.740428    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls podName:6c54280e-7b8b-4f95-bbe6-a93dd82018eb nodeName:}" failed. No retries permitted until 2026-11-02 07:23:59.240413593 +0000 UTC m=+30.209005848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-spr9r" (UID: "6c54280e-7b8b-4f95-bbe6-a93dd82018eb") : secret "samples-operator-tls" not found
Nov 02 07:23:58.740801 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.740481    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 07:23:58.740801 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:58.740509    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls podName:e3bc3d36-b8b2-409b-8c62-66590a8e3a5c nodeName:}" failed. No retries permitted until 2026-11-02 07:23:59.240500353 +0000 UTC m=+30.209092592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bbgm5" (UID: "e3bc3d36-b8b2-409b-8c62-66590a8e3a5c") : secret "cluster-monitoring-operator-tls" not found
Nov 02 07:23:58.740801 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.740641    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-tmp\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.741028 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.740835    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-config-volume\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:58.741154 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.741130    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:23:58.742357 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.742337    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/531c943e-bc09-4336-9d48-065b8b716dc4-serving-cert\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:23:58.742357 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.742347    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-serving-cert\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.742612 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.742592    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54a48a78-5f18-4e96-be7b-36f356cc9b39-cert\") pod \"ingress-canary-frv99\" (UID: \"54a48a78-5f18-4e96-be7b-36f356cc9b39\") " pod="openshift-ingress-canary/ingress-canary-frv99"
Nov 02 07:23:58.748292 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.748271    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zjjxl"
Nov 02 07:23:58.756959 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.756945    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w6xgl"
Nov 02 07:23:58.763681 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.763665    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx642\" (UniqueName: \"kubernetes.io/projected/7bf60630-1d98-4928-a2bf-dda7a1e1a110-kube-api-access-rx642\") pod \"network-check-source-6f78d59f99-lrtvc\" (UID: \"7bf60630-1d98-4928-a2bf-dda7a1e1a110\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc"
Nov 02 07:23:58.763974 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.763952    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh46m\" (UniqueName: \"kubernetes.io/projected/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-kube-api-access-qh46m\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:23:58.766519 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.766500    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbfdt\" (UniqueName: \"kubernetes.io/projected/531c943e-bc09-4336-9d48-065b8b716dc4-kube-api-access-vbfdt\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:23:58.766603 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.766535    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sd8w\" (UniqueName: \"kubernetes.io/projected/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-kube-api-access-9sd8w\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:23:58.766603 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.766544    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl4ww\" (UniqueName: \"kubernetes.io/projected/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-kube-api-access-jl4ww\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.766779 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.766677    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm5vh\" (UniqueName: \"kubernetes.io/projected/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-kube-api-access-bm5vh\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:58.767431 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.767409    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qj8\" (UniqueName: \"kubernetes.io/projected/54a48a78-5f18-4e96-be7b-36f356cc9b39-kube-api-access-k2qj8\") pod \"ingress-canary-frv99\" (UID: \"54a48a78-5f18-4e96-be7b-36f356cc9b39\") " pod="openshift-ingress-canary/ingress-canary-frv99"
Nov 02 07:23:58.775251 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.775232    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nfdng"
Nov 02 07:23:58.795931 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.795117    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc"
Nov 02 07:23:58.802256 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.802139    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:23:58.823329 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.823123    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:23:58.839397 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.837740    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:23:58.853805 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.853780    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:23:58.862332 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.862099    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc"
Nov 02 07:23:58.877132 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:58.876842    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-frv99"
Nov 02 07:23:59.069787 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.069762    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w6xgl"]
Nov 02 07:23:59.073456 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:59.073424    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437c4525_b041_4931_ba40_99c518193657.slice/crio-397c4955e3bebc429ee9892f20a9883bb475ad1f21c66f2b9433fb8f29f79bcc WatchSource:0}: Error finding container 397c4955e3bebc429ee9892f20a9883bb475ad1f21c66f2b9433fb8f29f79bcc: Status 404 returned error can't find the container with id 397c4955e3bebc429ee9892f20a9883bb475ad1f21c66f2b9433fb8f29f79bcc
Nov 02 07:23:59.075647 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.075623    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zjjxl"]
Nov 02 07:23:59.078485 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:59.078457    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f8157e_9211_4f25_886e_4a9e9338e730.slice/crio-8d4309e6ba2b04806fc412a1d6481add54d7a87f5e976ce51cb9a883a0c08124 WatchSource:0}: Error finding container 8d4309e6ba2b04806fc412a1d6481add54d7a87f5e976ce51cb9a883a0c08124: Status 404 returned error can't find the container with id 8d4309e6ba2b04806fc412a1d6481add54d7a87f5e976ce51cb9a883a0c08124
Nov 02 07:23:59.080924 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.080901    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nfdng"]
Nov 02 07:23:59.084274 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:59.084251    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85f18d7_a8b8_4634_af97_eb4e71dc5000.slice/crio-ee36e83bc2f2a81f701cf01449a70372e834c9d6115cc971a2193f2ffb154bac WatchSource:0}: Error finding container ee36e83bc2f2a81f701cf01449a70372e834c9d6115cc971a2193f2ffb154bac: Status 404 returned error can't find the container with id ee36e83bc2f2a81f701cf01449a70372e834c9d6115cc971a2193f2ffb154bac
Nov 02 07:23:59.127225 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.127194    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc"]
Nov 02 07:23:59.132070 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:59.132037    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd338565e_a9f0_44e8_ba6c_e0d88c253a10.slice/crio-500ef842aae7570570c53c39943b189bca7f232f753048216fd402c95d02ae62 WatchSource:0}: Error finding container 500ef842aae7570570c53c39943b189bca7f232f753048216fd402c95d02ae62: Status 404 returned error can't find the container with id 500ef842aae7570570c53c39943b189bca7f232f753048216fd402c95d02ae62
Nov 02 07:23:59.137673 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.137251    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"]
Nov 02 07:23:59.141774 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:59.141720    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ba5ec9_5277_460a_976e_3b1f154f7623.slice/crio-69753ffa641658976099a31f4bba3482713ecdfaf02065de78a88924eb0c3633 WatchSource:0}: Error finding container 69753ffa641658976099a31f4bba3482713ecdfaf02065de78a88924eb0c3633: Status 404 returned error can't find the container with id 69753ffa641658976099a31f4bba3482713ecdfaf02065de78a88924eb0c3633
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.142841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.142879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.142980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.143008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.143089    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.143266    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.143325    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:00.143306738 +0000 UTC m=+31.111898980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : secret "router-metrics-certs-default" not found
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.143477    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.143491    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649bdd4b4b-st5hm: secret "image-registry-tls" not found
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.143535    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls podName:124a32e6-1529-4b84-aa39-dda6f9332364 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:00.143522702 +0000 UTC m=+31.112114927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls") pod "image-registry-649bdd4b4b-st5hm" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364") : secret "image-registry-tls" not found
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.143591    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.143600    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769444d69f-x9wg9: secret "image-registry-tls" not found
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.143628    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls podName:9ff6ca66-e0d8-441f-8822-022929c12120 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:00.143618722 +0000 UTC m=+31.112210951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls") pod "image-registry-769444d69f-x9wg9" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120") : secret "image-registry-tls" not found
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.143709    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:00.143699676 +0000 UTC m=+31.112291901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : configmap references non-existent config key: service-ca.crt
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.143766    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 07:23:59.143824 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.143799    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert podName:06de30c9-de01-4f63-859a-23df97972617 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:00.143788269 +0000 UTC m=+31.112380495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wlj5v" (UID: "06de30c9-de01-4f63-859a-23df97972617") : secret "networking-console-plugin-cert" not found
Nov 02 07:23:59.163018 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.162960    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"]
Nov 02 07:23:59.169996 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:59.169962    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0196d04c_bd2b_48c7_bdb9_d31d83753247.slice/crio-524a3a58f9688d283291a8da385130fa4eece4ee1b65a366f303f8fd22de50bd WatchSource:0}: Error finding container 524a3a58f9688d283291a8da385130fa4eece4ee1b65a366f303f8fd22de50bd: Status 404 returned error can't find the container with id 524a3a58f9688d283291a8da385130fa4eece4ee1b65a366f303f8fd22de50bd
Nov 02 07:23:59.186401 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.186373    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tct4z"]
Nov 02 07:23:59.197752 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:59.197719    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af97e4d_867f_4a6c_8eb3_43d3ef4dce76.slice/crio-e6e2094399a94328715f9a2eff4342789fb49efbd39e8f6f40a4841d1f1c3f55 WatchSource:0}: Error finding container e6e2094399a94328715f9a2eff4342789fb49efbd39e8f6f40a4841d1f1c3f55: Status 404 returned error can't find the container with id e6e2094399a94328715f9a2eff4342789fb49efbd39e8f6f40a4841d1f1c3f55
Nov 02 07:23:59.215748 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.215565    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-frv99"]
Nov 02 07:23:59.222325 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.221459    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"]
Nov 02 07:23:59.225964 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.224572    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc"]
Nov 02 07:23:59.231664 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:23:59.231384    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf60630_1d98_4928_a2bf_dda7a1e1a110.slice/crio-acb44db5b2f90e52c9696bb3a11392dee0cdcd24b9635c8806a33259048792fd WatchSource:0}: Error finding container acb44db5b2f90e52c9696bb3a11392dee0cdcd24b9635c8806a33259048792fd: Status 404 returned error can't find the container with id acb44db5b2f90e52c9696bb3a11392dee0cdcd24b9635c8806a33259048792fd
Nov 02 07:23:59.243977 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.243949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:23:59.244090 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.244062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:23:59.244170 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.244072    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:23:59.244170 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.244137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:23:59.244170 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.244110    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 07:23:59.244316 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.244217    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls podName:6c54280e-7b8b-4f95-bbe6-a93dd82018eb nodeName:}" failed. No retries permitted until 2026-11-02 07:24:00.244187278 +0000 UTC m=+31.212779519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-spr9r" (UID: "6c54280e-7b8b-4f95-bbe6-a93dd82018eb") : secret "samples-operator-tls" not found
Nov 02 07:23:59.244316 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.244247    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls podName:08bc1af6-1c3b-4740-9dc0-fa309e83b78a nodeName:}" failed. No retries permitted until 2026-11-02 07:24:00.244240868 +0000 UTC m=+31.212833093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls") pod "dns-default-5tv7w" (UID: "08bc1af6-1c3b-4740-9dc0-fa309e83b78a") : secret "dns-default-metrics-tls" not found
Nov 02 07:23:59.244316 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.244273    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 07:23:59.244448 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:23:59.244342    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls podName:e3bc3d36-b8b2-409b-8c62-66590a8e3a5c nodeName:}" failed. No retries permitted until 2026-11-02 07:24:00.24431174 +0000 UTC m=+31.212903980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bbgm5" (UID: "e3bc3d36-b8b2-409b-8c62-66590a8e3a5c") : secret "cluster-monitoring-operator-tls" not found
Nov 02 07:23:59.876545 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.876505    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc" event={"ID":"7bf60630-1d98-4928-a2bf-dda7a1e1a110","Type":"ContainerStarted","Data":"acb44db5b2f90e52c9696bb3a11392dee0cdcd24b9635c8806a33259048792fd"}
Nov 02 07:23:59.879450 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.879353    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8" event={"ID":"531c943e-bc09-4336-9d48-065b8b716dc4","Type":"ContainerStarted","Data":"5d2612c2774448ed21ba7a86f46c5e7f2b47d9969c9a9f11654e50862790b3d1"}
Nov 02 07:23:59.881338 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.881312    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5" event={"ID":"0196d04c-bd2b-48c7-bdb9-d31d83753247","Type":"ContainerStarted","Data":"524a3a58f9688d283291a8da385130fa4eece4ee1b65a366f303f8fd22de50bd"}
Nov 02 07:23:59.883100 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.883076    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc" event={"ID":"d338565e-a9f0-44e8-ba6c-e0d88c253a10","Type":"ContainerStarted","Data":"500ef842aae7570570c53c39943b189bca7f232f753048216fd402c95d02ae62"}
Nov 02 07:23:59.885591 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.885556    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-frv99" event={"ID":"54a48a78-5f18-4e96-be7b-36f356cc9b39","Type":"ContainerStarted","Data":"dac0eec0e056dcff6da478c46b4c6500229124306772a1a808df1598c87e8a08"}
Nov 02 07:23:59.889560 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.889306    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nfdng" event={"ID":"f85f18d7-a8b8-4634-af97-eb4e71dc5000","Type":"ContainerStarted","Data":"24b051854aebc5ba35f2d5b01854f42ecc6f4afa5130a87851d6f5980887797e"}
Nov 02 07:23:59.889560 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.889344    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nfdng" event={"ID":"f85f18d7-a8b8-4634-af97-eb4e71dc5000","Type":"ContainerStarted","Data":"ee36e83bc2f2a81f701cf01449a70372e834c9d6115cc971a2193f2ffb154bac"}
Nov 02 07:23:59.891554 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.891529    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w6xgl" event={"ID":"437c4525-b041-4931-ba40-99c518193657","Type":"ContainerStarted","Data":"921ac9c309a9bda7633bbc15c0e0693fe586cef0bea212054f45c921f3a3390f"}
Nov 02 07:23:59.891643 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.891575    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w6xgl" event={"ID":"437c4525-b041-4931-ba40-99c518193657","Type":"ContainerStarted","Data":"397c4955e3bebc429ee9892f20a9883bb475ad1f21c66f2b9433fb8f29f79bcc"}
Nov 02 07:23:59.893881 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.893860    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zjjxl" event={"ID":"78f8157e-9211-4f25-886e-4a9e9338e730","Type":"ContainerStarted","Data":"7e5d09ac1292fcf06d3223ff531eeeaeba51dc127d3e31c907f499edca555c0f"}
Nov 02 07:23:59.893979 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.893913    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zjjxl" event={"ID":"78f8157e-9211-4f25-886e-4a9e9338e730","Type":"ContainerStarted","Data":"8d4309e6ba2b04806fc412a1d6481add54d7a87f5e976ce51cb9a883a0c08124"}
Nov 02 07:23:59.898181 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.898154    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tct4z" event={"ID":"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76","Type":"ContainerStarted","Data":"e6e2094399a94328715f9a2eff4342789fb49efbd39e8f6f40a4841d1f1c3f55"}
Nov 02 07:23:59.900527 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.900482    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh" event={"ID":"f4ba5ec9-5277-460a-976e-3b1f154f7623","Type":"ContainerStarted","Data":"69753ffa641658976099a31f4bba3482713ecdfaf02065de78a88924eb0c3633"}
Nov 02 07:23:59.931186 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.930551    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-nfdng" podStartSLOduration=381.930536296 podStartE2EDuration="6m21.930536296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:59.929440383 +0000 UTC m=+30.898032655" watchObservedRunningTime="2026-11-02 07:23:59.930536296 +0000 UTC m=+30.899128544"
Nov 02 07:23:59.971455 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:23:59.971277    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zjjxl" podStartSLOduration=381.971259628 podStartE2EDuration="6m21.971259628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:23:59.969435292 +0000 UTC m=+30.938027540" watchObservedRunningTime="2026-11-02 07:23:59.971259628 +0000 UTC m=+30.939851875"
Nov 02 07:24:00.021358 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:00.020631    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-w6xgl" podStartSLOduration=382.020615359 podStartE2EDuration="6m22.020615359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:00.019197533 +0000 UTC m=+30.987789783" watchObservedRunningTime="2026-11-02 07:24:00.020615359 +0000 UTC m=+30.989207606"
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:00.156512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:00.156622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:00.156834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:00.156949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:00.157088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.157363    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.157528    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:02.157502809 +0000 UTC m=+33.126095049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : secret "router-metrics-certs-default" not found
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.158215    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.158232    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649bdd4b4b-st5hm: secret "image-registry-tls" not found
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.158279    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls podName:124a32e6-1529-4b84-aa39-dda6f9332364 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:02.158263401 +0000 UTC m=+33.126855640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls") pod "image-registry-649bdd4b4b-st5hm" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364") : secret "image-registry-tls" not found
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.158339    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.158349    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769444d69f-x9wg9: secret "image-registry-tls" not found
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.158378    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls podName:9ff6ca66-e0d8-441f-8822-022929c12120 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:02.158367944 +0000 UTC m=+33.126960185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls") pod "image-registry-769444d69f-x9wg9" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120") : secret "image-registry-tls" not found
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.158441    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:02.158430805 +0000 UTC m=+33.127023045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : configmap references non-existent config key: service-ca.crt
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.158496    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 07:24:00.158562 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.158531    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert podName:06de30c9-de01-4f63-859a-23df97972617 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:02.158520545 +0000 UTC m=+33.127112784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wlj5v" (UID: "06de30c9-de01-4f63-859a-23df97972617") : secret "networking-console-plugin-cert" not found
Nov 02 07:24:00.259107 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:00.258144    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:00.259107 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:00.258281    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:00.259107 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:00.258320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:00.259107 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.258494    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 07:24:00.259107 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.258551    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls podName:e3bc3d36-b8b2-409b-8c62-66590a8e3a5c nodeName:}" failed. No retries permitted until 2026-11-02 07:24:02.258534177 +0000 UTC m=+33.227126415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bbgm5" (UID: "e3bc3d36-b8b2-409b-8c62-66590a8e3a5c") : secret "cluster-monitoring-operator-tls" not found
Nov 02 07:24:00.259107 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.258939    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:24:00.259107 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.258989    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls podName:08bc1af6-1c3b-4740-9dc0-fa309e83b78a nodeName:}" failed. No retries permitted until 2026-11-02 07:24:02.258974067 +0000 UTC m=+33.227566307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls") pod "dns-default-5tv7w" (UID: "08bc1af6-1c3b-4740-9dc0-fa309e83b78a") : secret "dns-default-metrics-tls" not found
Nov 02 07:24:00.259107 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.259044    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 07:24:00.259107 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:00.259082    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls podName:6c54280e-7b8b-4f95-bbe6-a93dd82018eb nodeName:}" failed. No retries permitted until 2026-11-02 07:24:02.259070849 +0000 UTC m=+33.227663078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-spr9r" (UID: "6c54280e-7b8b-4f95-bbe6-a93dd82018eb") : secret "samples-operator-tls" not found
Nov 02 07:24:01.914781 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:01.914707    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc" event={"ID":"d338565e-a9f0-44e8-ba6c-e0d88c253a10","Type":"ContainerStarted","Data":"7d940e89f704785ecb2d7f24781751379dc9daf8881a11b424777755545ec3af"}
Nov 02 07:24:01.957048 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:01.955718    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc" podStartSLOduration=355.782133309 podStartE2EDuration="5m57.955696506s" totalImagesPullingTime="2.173563197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:01.954947846 +0000 UTC m=+32.923540094" watchObservedRunningTime="2026-11-02 07:24:01.955696506 +0000 UTC m=+32.924288755"
Nov 02 07:24:02.178844 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:02.178757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:02.178844 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:02.178818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:24:02.179163 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.179042    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.179022869 +0000 UTC m=+37.147615115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : configmap references non-existent config key: service-ca.crt
Nov 02 07:24:02.179163 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.179139    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 07:24:02.179387 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.179210    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert podName:06de30c9-de01-4f63-859a-23df97972617 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.179192181 +0000 UTC m=+37.147784420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wlj5v" (UID: "06de30c9-de01-4f63-859a-23df97972617") : secret "networking-console-plugin-cert" not found
Nov 02 07:24:02.179444 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:02.179425    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:02.179575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:02.179504    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:02.179575 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:02.179533    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:02.179575 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.179570    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 07:24:02.179705 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.179613    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.179602146 +0000 UTC m=+37.148194386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : secret "router-metrics-certs-default" not found
Nov 02 07:24:02.179705 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.179648    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:02.179705 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.179656    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:02.179705 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.179661    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769444d69f-x9wg9: secret "image-registry-tls" not found
Nov 02 07:24:02.179705 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.179669    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649bdd4b4b-st5hm: secret "image-registry-tls" not found
Nov 02 07:24:02.179705 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.179714    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls podName:9ff6ca66-e0d8-441f-8822-022929c12120 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.179698937 +0000 UTC m=+37.148291184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls") pod "image-registry-769444d69f-x9wg9" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120") : secret "image-registry-tls" not found
Nov 02 07:24:02.179705 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.179733    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls podName:124a32e6-1529-4b84-aa39-dda6f9332364 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.179723635 +0000 UTC m=+37.148315894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls") pod "image-registry-649bdd4b4b-st5hm" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364") : secret "image-registry-tls" not found
Nov 02 07:24:02.280680 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:02.280350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:02.280680 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:02.280438    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:02.280680 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:02.280464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:02.280680 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.280552    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:24:02.280680 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.280575    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 07:24:02.280680 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.280616    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 07:24:02.280680 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.280630    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls podName:08bc1af6-1c3b-4740-9dc0-fa309e83b78a nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.280614827 +0000 UTC m=+37.249207051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls") pod "dns-default-5tv7w" (UID: "08bc1af6-1c3b-4740-9dc0-fa309e83b78a") : secret "dns-default-metrics-tls" not found
Nov 02 07:24:02.280680 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.280645    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls podName:6c54280e-7b8b-4f95-bbe6-a93dd82018eb nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.280639254 +0000 UTC m=+37.249231480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-spr9r" (UID: "6c54280e-7b8b-4f95-bbe6-a93dd82018eb") : secret "samples-operator-tls" not found
Nov 02 07:24:02.280680 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:02.280658    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls podName:e3bc3d36-b8b2-409b-8c62-66590a8e3a5c nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.280650153 +0000 UTC m=+37.249242379 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bbgm5" (UID: "e3bc3d36-b8b2-409b-8c62-66590a8e3a5c") : secret "cluster-monitoring-operator-tls" not found
Nov 02 07:24:03.293333 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.293267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:24:03.293667 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.293399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:24:03.293667 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:03.293489    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 07:24:03.293667 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:03.293534    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs podName:0059120b-d9ca-462c-a74b-bb99ea44c3d7 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:35.293520837 +0000 UTC m=+66.262113063 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs") pod "network-metrics-daemon-7j98p" (UID: "0059120b-d9ca-462c-a74b-bb99ea44c3d7") : secret "metrics-daemon-secret" not found
Nov 02 07:24:03.304861 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.303529    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:24:03.394060 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.394028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjhf\" (UniqueName: \"kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf\") pod \"network-check-target-g88mp\" (UID: \"c90ce63c-e5e9-4edf-8a1c-78ee6d092633\") " pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:24:03.397251 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.397227    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:24:03.411714 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.411662    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crjhf\" (UniqueName: \"kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf\") pod \"network-check-target-g88mp\" (UID: \"c90ce63c-e5e9-4edf-8a1c-78ee6d092633\") " pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:24:03.708101 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.708072    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:24:03.756272 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.756215    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rs7kb"]
Nov 02 07:24:03.819271 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:24:03.818483    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb5657a_bb11_4279_86f6_35118656d990.slice/crio-0c70c196a72bdf35fcd10843918f22e32bc3a6a546042d67faa1db8244cf97c6 WatchSource:0}: Error finding container 0c70c196a72bdf35fcd10843918f22e32bc3a6a546042d67faa1db8244cf97c6: Status 404 returned error can't find the container with id 0c70c196a72bdf35fcd10843918f22e32bc3a6a546042d67faa1db8244cf97c6
Nov 02 07:24:03.932955 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.930837    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rs7kb" event={"ID":"9bb5657a-bb11-4279-86f6-35118656d990","Type":"ContainerStarted","Data":"0c70c196a72bdf35fcd10843918f22e32bc3a6a546042d67faa1db8244cf97c6"}
Nov 02 07:24:03.934409 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.934375    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tct4z" event={"ID":"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76","Type":"ContainerStarted","Data":"57b2b86c17341018899b22ce895095e90fc7aedd4c5dc746a09267f74172471f"}
Nov 02 07:24:03.936803 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.936779    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/0.log"
Nov 02 07:24:03.936952 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.936819    2576 generic.go:356] "Generic (PLEG): container finished" podID="f4ba5ec9-5277-460a-976e-3b1f154f7623" containerID="726bd4447b708365932eaec9b5ec00b988c1ad3226e16dbe8cdd6e7f6e43cfc4" exitCode=255
Nov 02 07:24:03.936952 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.936862    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh" event={"ID":"f4ba5ec9-5277-460a-976e-3b1f154f7623","Type":"ContainerDied","Data":"726bd4447b708365932eaec9b5ec00b988c1ad3226e16dbe8cdd6e7f6e43cfc4"}
Nov 02 07:24:03.937378 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.937109    2576 scope.go:122] "RemoveContainer" containerID="726bd4447b708365932eaec9b5ec00b988c1ad3226e16dbe8cdd6e7f6e43cfc4"
Nov 02 07:24:03.938664 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.938643    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5" event={"ID":"0196d04c-bd2b-48c7-bdb9-d31d83753247","Type":"ContainerStarted","Data":"dc236a499f5c5e55d2bf536ff40f91f02f091590ff5239a280b81ee408f6cf69"}
Nov 02 07:24:03.977034 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:03.976988    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-tct4z" podStartSLOduration=360.787373232 podStartE2EDuration="6m4.976973666s" totalImagesPullingTime="4.189600434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:03.975562897 +0000 UTC m=+34.944155156" watchObservedRunningTime="2026-11-02 07:24:03.976973666 +0000 UTC m=+34.945565908"
Nov 02 07:24:04.045808 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.043448    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g88mp"]
Nov 02 07:24:04.057109 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.057060    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5" podStartSLOduration=361.55804387 podStartE2EDuration="6m6.057047144s" totalImagesPullingTime="4.499003274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:04.056265485 +0000 UTC m=+35.024857732" watchObservedRunningTime="2026-11-02 07:24:04.057047144 +0000 UTC m=+35.025639411"
Nov 02 07:24:04.183607 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:24:04.183575    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90ce63c_e5e9_4edf_8a1c_78ee6d092633.slice/crio-932b6142154dd73da54393785d8b89ba5d58709c497f2b30dc8573442ee1b10a WatchSource:0}: Error finding container 932b6142154dd73da54393785d8b89ba5d58709c497f2b30dc8573442ee1b10a: Status 404 returned error can't find the container with id 932b6142154dd73da54393785d8b89ba5d58709c497f2b30dc8573442ee1b10a
Nov 02 07:24:04.745920 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.745498    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vhjhr"]
Nov 02 07:24:04.768074 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.767547    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:04.772378 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.772307    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vhjhr"]
Nov 02 07:24:04.782345 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.781826    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 07:24:04.782345 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.781871    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jc9nv\""
Nov 02 07:24:04.782761 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.782743    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 07:24:04.910587 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.910566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:04.910637 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.910601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spj5p\" (UniqueName: \"kubernetes.io/projected/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-api-access-spj5p\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:04.910754 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.910729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:04.910866 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.910768    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb81587c-bf28-4f77-ac53-02cc794c5365-data-volume\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:04.910942 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.910922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bb81587c-bf28-4f77-ac53-02cc794c5365-crio-socket\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:04.943752 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.943722    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8" event={"ID":"531c943e-bc09-4336-9d48-065b8b716dc4","Type":"ContainerStarted","Data":"b274e39dd2baa0fa2c4843886fcc89b04b834b10dc440ad51ed07c3cdb0edf4f"}
Nov 02 07:24:04.945364 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.945323    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-frv99" event={"ID":"54a48a78-5f18-4e96-be7b-36f356cc9b39","Type":"ContainerStarted","Data":"260a9bccd437cd1743a5d0a39aaa55f0dcf0e8c1c4716cfec6aa5157f4cee973"}
Nov 02 07:24:04.947704 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.947625    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/1.log"
Nov 02 07:24:04.948470 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.948450    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/0.log"
Nov 02 07:24:04.948566 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.948488    2576 generic.go:356] "Generic (PLEG): container finished" podID="f4ba5ec9-5277-460a-976e-3b1f154f7623" containerID="9d164f45337214cb364492d85495747f3847fda7adc1f6f41a76587105e0971b" exitCode=255
Nov 02 07:24:04.948566 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.948555    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh" event={"ID":"f4ba5ec9-5277-460a-976e-3b1f154f7623","Type":"ContainerDied","Data":"9d164f45337214cb364492d85495747f3847fda7adc1f6f41a76587105e0971b"}
Nov 02 07:24:04.948664 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.948596    2576 scope.go:122] "RemoveContainer" containerID="726bd4447b708365932eaec9b5ec00b988c1ad3226e16dbe8cdd6e7f6e43cfc4"
Nov 02 07:24:04.948996 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.948968    2576 scope.go:122] "RemoveContainer" containerID="9d164f45337214cb364492d85495747f3847fda7adc1f6f41a76587105e0971b"
Nov 02 07:24:04.949165 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:04.949143    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tqzjh_openshift-console-operator(f4ba5ec9-5277-460a-976e-3b1f154f7623)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh" podUID="f4ba5ec9-5277-460a-976e-3b1f154f7623"
Nov 02 07:24:04.952807 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.952782    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g88mp" event={"ID":"c90ce63c-e5e9-4edf-8a1c-78ee6d092633","Type":"ContainerStarted","Data":"932b6142154dd73da54393785d8b89ba5d58709c497f2b30dc8573442ee1b10a"}
Nov 02 07:24:04.987864 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:04.987811    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8" podStartSLOduration=356.998709271 podStartE2EDuration="6m1.987796978s" totalImagesPullingTime="4.989087707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:04.983518346 +0000 UTC m=+35.952110594" watchObservedRunningTime="2026-11-02 07:24:04.987796978 +0000 UTC m=+35.956389227"
Nov 02 07:24:05.013468 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.011661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:05.013468 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.011709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb81587c-bf28-4f77-ac53-02cc794c5365-data-volume\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:05.013468 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.011775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bb81587c-bf28-4f77-ac53-02cc794c5365-crio-socket\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:05.013468 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.011876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:05.013929 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.013903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spj5p\" (UniqueName: \"kubernetes.io/projected/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-api-access-spj5p\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:05.014038 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.013827    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bb81587c-bf28-4f77-ac53-02cc794c5365-crio-socket\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:05.014038 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:05.013780    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:05.014157 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:05.014045    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls podName:bb81587c-bf28-4f77-ac53-02cc794c5365 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:05.514022827 +0000 UTC m=+36.482615053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhjhr" (UID: "bb81587c-bf28-4f77-ac53-02cc794c5365") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:05.014307 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:05.013829    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 07:24:05.014374 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:05.014361    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm podName:bb81587c-bf28-4f77-ac53-02cc794c5365 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:05.514334179 +0000 UTC m=+36.482926406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhjhr" (UID: "bb81587c-bf28-4f77-ac53-02cc794c5365") : configmap "kube-rbac-proxy" not found
Nov 02 07:24:05.014431 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.014381    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb81587c-bf28-4f77-ac53-02cc794c5365-data-volume\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:05.026926 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.026218    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-frv99" podStartSLOduration=1.696551065 podStartE2EDuration="7.026199099s" totalImagesPullingTime="5.329648034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:05.025149788 +0000 UTC m=+35.993742035" watchObservedRunningTime="2026-11-02 07:24:05.026199099 +0000 UTC m=+35.994791347"
Nov 02 07:24:05.045933 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.045856    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spj5p\" (UniqueName: \"kubernetes.io/projected/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-api-access-spj5p\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:05.061091 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.060975    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8"]
Nov 02 07:24:05.077718 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.077694    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8"
Nov 02 07:24:05.100634 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.100610    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8"]
Nov 02 07:24:05.101640 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.101606    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:05.101844 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.101747    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 07:24:05.122319 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.122289    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-48668\""
Nov 02 07:24:05.215814 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.215726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fs4t\" (UniqueName: \"kubernetes.io/projected/7886c1fa-91a5-4ac1-9c59-98a8268bb05e-kube-api-access-9fs4t\") pod \"migrator-66df5d496b-p7hb8\" (UID: \"7886c1fa-91a5-4ac1-9c59-98a8268bb05e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8"
Nov 02 07:24:05.317154 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.317125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fs4t\" (UniqueName: \"kubernetes.io/projected/7886c1fa-91a5-4ac1-9c59-98a8268bb05e-kube-api-access-9fs4t\") pod \"migrator-66df5d496b-p7hb8\" (UID: \"7886c1fa-91a5-4ac1-9c59-98a8268bb05e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8"
Nov 02 07:24:05.341722 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.341698    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fs4t\" (UniqueName: \"kubernetes.io/projected/7886c1fa-91a5-4ac1-9c59-98a8268bb05e-kube-api-access-9fs4t\") pod \"migrator-66df5d496b-p7hb8\" (UID: \"7886c1fa-91a5-4ac1-9c59-98a8268bb05e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8"
Nov 02 07:24:05.416662 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.416633    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8"
Nov 02 07:24:05.519306 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.519280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:05.519422 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.519381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:05.519467 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:05.519452    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 07:24:05.519502 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:05.519498    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm podName:bb81587c-bf28-4f77-ac53-02cc794c5365 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.519485373 +0000 UTC m=+37.488077599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhjhr" (UID: "bb81587c-bf28-4f77-ac53-02cc794c5365") : configmap "kube-rbac-proxy" not found
Nov 02 07:24:05.520086 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:05.519793    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:05.520086 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:05.519829    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls podName:bb81587c-bf28-4f77-ac53-02cc794c5365 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:06.519820046 +0000 UTC m=+37.488412271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhjhr" (UID: "bb81587c-bf28-4f77-ac53-02cc794c5365") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:05.732473 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.732437    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8"]
Nov 02 07:24:05.738532 ip-10-0-134-79 kubenswrapper[2576]: W1102 07:24:05.738461    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7886c1fa_91a5_4ac1_9c59_98a8268bb05e.slice/crio-b199a2a5e18e5bb1c841f6d0a38cee1d6f7e82f13c39bea903ce5d017916933b WatchSource:0}: Error finding container b199a2a5e18e5bb1c841f6d0a38cee1d6f7e82f13c39bea903ce5d017916933b: Status 404 returned error can't find the container with id b199a2a5e18e5bb1c841f6d0a38cee1d6f7e82f13c39bea903ce5d017916933b
Nov 02 07:24:05.957206 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.957179    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/1.log"
Nov 02 07:24:05.957595 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.957497    2576 scope.go:122] "RemoveContainer" containerID="9d164f45337214cb364492d85495747f3847fda7adc1f6f41a76587105e0971b"
Nov 02 07:24:05.957736 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:05.957710    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tqzjh_openshift-console-operator(f4ba5ec9-5277-460a-976e-3b1f154f7623)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh" podUID="f4ba5ec9-5277-460a-976e-3b1f154f7623"
Nov 02 07:24:05.958824 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.958799    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g88mp" event={"ID":"c90ce63c-e5e9-4edf-8a1c-78ee6d092633","Type":"ContainerStarted","Data":"3665c0658529e16355409350289a2372fc5c863892401f26ccdb1865509d01d7"}
Nov 02 07:24:05.958947 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.958851    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:24:05.959936 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.959916    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8" event={"ID":"7886c1fa-91a5-4ac1-9c59-98a8268bb05e","Type":"ContainerStarted","Data":"b199a2a5e18e5bb1c841f6d0a38cee1d6f7e82f13c39bea903ce5d017916933b"}
Nov 02 07:24:05.961265 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:05.961238    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc" event={"ID":"7bf60630-1d98-4928-a2bf-dda7a1e1a110","Type":"ContainerStarted","Data":"d77fbbb0988e0528ea7ce39d815b96f8dca590e29c6aac9407ef929461aa6dad"}
Nov 02 07:24:06.044941 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:06.044810    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g88mp" podStartSLOduration=35.346466797 podStartE2EDuration="36.044792345s" totalImagesPullingTime="698.325548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:06.043520114 +0000 UTC m=+37.012112360" watchObservedRunningTime="2026-11-02 07:24:06.044792345 +0000 UTC m=+37.013384595"
Nov 02 07:24:06.083215 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:06.083161    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc" podStartSLOduration=333.410771501 podStartE2EDuration="5m39.083145181s" totalImagesPullingTime="5.67237368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:06.082389404 +0000 UTC m=+37.050981650" watchObservedRunningTime="2026-11-02 07:24:06.083145181 +0000 UTC m=+37.051737429"
Nov 02 07:24:06.227966 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:06.227934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:06.227966 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:06.227967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:06.228203 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:06.228026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:06.228203 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:06.228046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:24:06.228203 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.228090    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:06.228203 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.228113    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649bdd4b4b-st5hm: secret "image-registry-tls" not found
Nov 02 07:24:06.228203 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.228160    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 07:24:06.228203 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.228188    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls podName:124a32e6-1529-4b84-aa39-dda6f9332364 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.228167606 +0000 UTC m=+45.196759848 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls") pod "image-registry-649bdd4b4b-st5hm" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364") : secret "image-registry-tls" not found
Nov 02 07:24:06.228383 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.228218    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert podName:06de30c9-de01-4f63-859a-23df97972617 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.228204223 +0000 UTC m=+45.196796468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wlj5v" (UID: "06de30c9-de01-4f63-859a-23df97972617") : secret "networking-console-plugin-cert" not found
Nov 02 07:24:06.228383 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:06.228252    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:06.228383 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.228277    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.228264397 +0000 UTC m=+45.196856637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : configmap references non-existent config key: service-ca.crt
Nov 02 07:24:06.228383 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.228309    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 07:24:06.228383 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.228165    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:06.228383 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.228341    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.228330325 +0000 UTC m=+45.196922567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : secret "router-metrics-certs-default" not found
Nov 02 07:24:06.228383 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.228340    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769444d69f-x9wg9: secret "image-registry-tls" not found
Nov 02 07:24:06.228383 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.228381    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls podName:9ff6ca66-e0d8-441f-8822-022929c12120 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.228371096 +0000 UTC m=+45.196963356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls") pod "image-registry-769444d69f-x9wg9" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120") : secret "image-registry-tls" not found
Nov 02 07:24:06.328980 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:06.328885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:06.329125 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:06.328985    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:06.329125 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:06.329009    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:06.329243 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.329139    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 07:24:06.329243 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.329188    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls podName:e3bc3d36-b8b2-409b-8c62-66590a8e3a5c nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.329175642 +0000 UTC m=+45.297767868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bbgm5" (UID: "e3bc3d36-b8b2-409b-8c62-66590a8e3a5c") : secret "cluster-monitoring-operator-tls" not found
Nov 02 07:24:06.329508 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.329489    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:24:06.329559 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.329546    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls podName:08bc1af6-1c3b-4740-9dc0-fa309e83b78a nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.329531745 +0000 UTC m=+45.298123975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls") pod "dns-default-5tv7w" (UID: "08bc1af6-1c3b-4740-9dc0-fa309e83b78a") : secret "dns-default-metrics-tls" not found
Nov 02 07:24:06.329620 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.329609    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 07:24:06.329675 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.329640    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls podName:6c54280e-7b8b-4f95-bbe6-a93dd82018eb nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.329629877 +0000 UTC m=+45.298222107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-spr9r" (UID: "6c54280e-7b8b-4f95-bbe6-a93dd82018eb") : secret "samples-operator-tls" not found
Nov 02 07:24:06.531574 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:06.531209    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:06.531574 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:06.531325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:06.531574 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.531364    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 07:24:06.531574 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.531445    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm podName:bb81587c-bf28-4f77-ac53-02cc794c5365 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:08.531424582 +0000 UTC m=+39.500016807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhjhr" (UID: "bb81587c-bf28-4f77-ac53-02cc794c5365") : configmap "kube-rbac-proxy" not found
Nov 02 07:24:06.531574 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.531500    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:06.531574 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:06.531546    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls podName:bb81587c-bf28-4f77-ac53-02cc794c5365 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:08.53153046 +0000 UTC m=+39.500122685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhjhr" (UID: "bb81587c-bf28-4f77-ac53-02cc794c5365") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:07.969875 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:07.969838    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8" event={"ID":"7886c1fa-91a5-4ac1-9c59-98a8268bb05e","Type":"ContainerStarted","Data":"8ae1a7157df54a1f33176dbe2c1aa6cfe08797dd9e23e857752539c939853191"}
Nov 02 07:24:08.011044 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:08.010983    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8" podStartSLOduration=1.447864717 podStartE2EDuration="3.010961789s" totalImagesPullingTime="1.563097072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:08.009675346 +0000 UTC m=+38.978267596" watchObservedRunningTime="2026-11-02 07:24:08.010961789 +0000 UTC m=+38.979554036"
Nov 02 07:24:08.548599 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:08.548559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:08.548767 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:08.548650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:08.548767 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:08.548683    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 07:24:08.548767 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:08.548749    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm podName:bb81587c-bf28-4f77-ac53-02cc794c5365 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.548734777 +0000 UTC m=+43.517327003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhjhr" (UID: "bb81587c-bf28-4f77-ac53-02cc794c5365") : configmap "kube-rbac-proxy" not found
Nov 02 07:24:08.548902 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:08.548782    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:08.548902 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:08.548828    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls podName:bb81587c-bf28-4f77-ac53-02cc794c5365 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.548817491 +0000 UTC m=+43.517409717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhjhr" (UID: "bb81587c-bf28-4f77-ac53-02cc794c5365") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:08.802949 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:08.802857    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:08.802949 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:08.802927    2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:08.803246 ip-10-0-134-79 kubenswrapper[2576]: I1102 07:24:08.803234    2576 scope.go:122] "RemoveContainer" containerID="9d164f45337214cb364492d85495747f3847fda7adc1f6f41a76587105e0971b"
Nov 02 07:24:08.803420 ip-10-0-134-79 kubenswrapper[2576]: E1102 07:24:08.803404    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tqzjh_openshift-console-operator(f4ba5ec9-5277-460a-976e-3b1f154f7623)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh" podUID="f4ba5ec9-5277-460a-976e-3b1f154f7623"
Nov 02 07:24:09.238149 ip-10-0-134-79 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 07:24:09.255250 ip-10-0-134-79 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 07:24:09.255413 ip-10-0-134-79 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 07:24:09.255578 ip-10-0-134-79 systemd[1]: kubelet.service: Consumed 2.930s CPU time.
Nov 02 07:24:09.267656 ip-10-0-134-79 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 07:24:09.375495 ip-10-0-134-79 kubenswrapper[4898]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 07:24:09.375495 ip-10-0-134-79 kubenswrapper[4898]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 07:24:09.375495 ip-10-0-134-79 kubenswrapper[4898]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 07:24:09.375495 ip-10-0-134-79 kubenswrapper[4898]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 07:24:09.378215 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378133    4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:09.378215 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378213    4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:09.378215 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378219    4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378222    4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378225    4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378228    4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378230    4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378233    4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378236    4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378238    4898 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378241    4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378244    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378246    4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378249    4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378252    4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378254    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378257    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378259    4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378262    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378264    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378267    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378270    4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:09.378344 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378272    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378275    4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378277    4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378280    4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378283    4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378286    4898 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378296    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378299    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378302    4898 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378304    4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378307    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378309    4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378312    4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378314    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378317    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378319    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378322    4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378324    4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378328    4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378333    4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:09.378851 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378336    4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378339    4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378342    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378345    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378348    4898 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378351    4898 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378354    4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378357    4898 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378360    4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378362    4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378365    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378368    4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378371    4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378373    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378377    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378379    4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378382    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378384    4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378387    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378390    4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:09.379376 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378392    4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378395    4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378397    4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378400    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378403    4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378405    4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378408    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378411    4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378414    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378417    4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378419    4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378422    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378424    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378427    4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378429    4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378432    4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378435    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378438    4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378440    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:09.379883 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378443    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378446    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378448    4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378451    4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378453    4898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378456    4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378461    4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378464    4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378467    4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378470    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378473    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378475    4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378478    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378480    4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378483    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378486    4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378488    4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378491    4898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378494    4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:09.380390 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378497    4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378500    4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378502    4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378505    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378507    4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.378510    4898 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379273    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379281    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379283    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379287    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379290    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379293    4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379296    4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379298    4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379302    4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379306    4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379310    4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379313    4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379316    4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:09.380936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379320    4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379323    4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379326    4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379329    4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379332    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379335    4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379338    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379341    4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379344    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379346    4898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379349    4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379352    4898 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379355    4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379357    4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379360    4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379363    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379365    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379368    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379370    4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379374    4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:09.381420 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379376    4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379379    4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379383    4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379385    4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379388    4898 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379391    4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379394    4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379397    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379400    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379403    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379406    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379408    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379411    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379413    4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379416    4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379418    4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379421    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379423    4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379426    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:09.381929 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379430    4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379434    4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379437    4898 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379440    4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379443    4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379447    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379450    4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379454    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379456    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379459    4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379462    4898 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379464    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379470    4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379473    4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379476    4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379478    4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379481    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379484    4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379486    4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379489    4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:09.382426 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379492    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379495    4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379498    4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379500    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379503    4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379505    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379508    4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379511    4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379513    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379516    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379518    4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379521    4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379523    4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379526    4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379529    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379531    4898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379534    4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379537    4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379539    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379542    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:09.382957 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379545    4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379548    4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379551    4898 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379554    4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379558    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379560    4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379563    4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379566    4898 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379568    4898 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379571    4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379573    4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379576    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379578    4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.379581    4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380074    4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380080    4898 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380083    4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380086    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380089    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380092    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:09.383457 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380094    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380097    4898 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380100    4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380102    4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380105    4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380108    4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380111    4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380113    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380116    4898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380118    4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380121    4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380124    4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380126    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380129    4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380132    4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380134    4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380139    4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380142    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380145    4898 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380148    4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:09.383996 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380150    4898 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380153    4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380155    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380158    4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380161    4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380163    4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380166    4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380169    4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380171    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380174    4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380177    4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380180    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380182    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380184    4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380187    4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380189    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380192    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380194    4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380197    4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380199    4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:09.384504 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380209    4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380211    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380214    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380217    4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380221    4898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380224    4898 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380226    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380229    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380234    4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380237    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380239    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380242    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380244    4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380247    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380249    4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380252    4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380255    4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380257    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380260    4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:09.385027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380263    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380266    4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380269    4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380271    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380274    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380276    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380279    4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380281    4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380284    4898 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380286    4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380289    4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380291    4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380297    4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380300    4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380302    4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380305    4898 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380307    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380310    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380313    4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:09.385524 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380315    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380318    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380322    4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380324    4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380327    4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380329    4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380332    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380334    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380339    4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380343    4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380347    4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380350    4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380352    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380355    4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380357    4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380360    4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380363    4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380366    4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380368    4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:09.386008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380371    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380373    4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.380377    4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380472    4898 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380486    4898 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380493    4898 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380501    4898 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380508    4898 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380511    4898 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380516    4898 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380520    4898 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380523    4898 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380526    4898 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380530    4898 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380533    4898 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380537    4898 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380540    4898 flags.go:64] FLAG: --cgroup-root=""
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380543    4898 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380547    4898 flags.go:64] FLAG: --client-ca-file=""
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380550    4898 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380552    4898 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380557    4898 flags.go:64] FLAG: --cluster-domain=""
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380560    4898 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380563    4898 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 07:24:09.386489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380566    4898 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380569    4898 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380573    4898 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380576    4898 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380579    4898 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380583    4898 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380586    4898 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380589    4898 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380593    4898 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380596    4898 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380599    4898 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380603    4898 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380606    4898 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380610    4898 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380614    4898 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380617    4898 flags.go:64] FLAG: --enable-server="true"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380620    4898 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380625    4898 flags.go:64] FLAG: --event-burst="100"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380628    4898 flags.go:64] FLAG: --event-qps="50"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380631    4898 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380634    4898 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380637    4898 flags.go:64] FLAG: --eviction-hard=""
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380642    4898 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380645    4898 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380649    4898 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 07:24:09.387128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380652    4898 flags.go:64] FLAG: --eviction-soft=""
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380655    4898 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380658    4898 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380661    4898 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380664    4898 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380667    4898 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380670    4898 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380673    4898 flags.go:64] FLAG: --feature-gates=""
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380677    4898 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380680    4898 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380684    4898 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380687    4898 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380690    4898 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380694    4898 flags.go:64] FLAG: --help="false"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380697    4898 flags.go:64] FLAG: --hostname-override="ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380700    4898 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380703    4898 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380706    4898 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380709    4898 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380713    4898 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380716    4898 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380720    4898 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380723    4898 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 07:24:09.387767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380726    4898 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380729    4898 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380732    4898 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380735    4898 flags.go:64] FLAG: --kube-reserved=""
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380738    4898 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380741    4898 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380744    4898 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380747    4898 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380750    4898 flags.go:64] FLAG: --lock-file=""
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380753    4898 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380756    4898 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380759    4898 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380764    4898 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380767    4898 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380770    4898 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380773    4898 flags.go:64] FLAG: --logging-format="text"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380776    4898 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380779    4898 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380782    4898 flags.go:64] FLAG: --manifest-url=""
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380785    4898 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380789    4898 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380793    4898 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380797    4898 flags.go:64] FLAG: --max-pods="110"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380800    4898 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380803    4898 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 07:24:09.388371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380806    4898 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380809    4898 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380812    4898 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380815    4898 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380818    4898 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380826    4898 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380830    4898 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380833    4898 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380836    4898 flags.go:64] FLAG: --pod-cidr=""
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380839    4898 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380842    4898 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380845    4898 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380848    4898 flags.go:64] FLAG: --port="10250"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380851    4898 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380854    4898 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0166363acc16366c0"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380858    4898 flags.go:64] FLAG: --qos-reserved=""
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380860    4898 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380864    4898 flags.go:64] FLAG: --register-node="true"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380867    4898 flags.go:64] FLAG: --register-with-taints=""
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380871    4898 flags.go:64] FLAG: --registry-burst="10"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380874    4898 flags.go:64] FLAG: --registry-qps="5"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380877    4898 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380879    4898 flags.go:64] FLAG: --reserved-memory=""
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380883    4898 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380899    4898 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380902    4898 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 07:24:09.389013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380905    4898 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380908    4898 flags.go:64] FLAG: --runonce="false"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380911    4898 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380915    4898 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380918    4898 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380921    4898 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380924    4898 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380929    4898 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380932    4898 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380935    4898 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380938    4898 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380943    4898 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380946    4898 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380949    4898 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380952    4898 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380956    4898 flags.go:64] FLAG: --system-cgroups=""
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380958    4898 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380964    4898 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380967    4898 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380970    4898 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380973    4898 flags.go:64] FLAG: --tls-min-version=""
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380976    4898 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380979    4898 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380982    4898 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380985    4898 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 07:24:09.389644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380988    4898 flags.go:64] FLAG: --v="2"
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380992    4898 flags.go:64] FLAG: --version="false"
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380996    4898 flags.go:64] FLAG: --vmodule=""
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.380999    4898 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.381003    4898 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381133    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381137    4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381140    4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381143    4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381145    4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381148    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381151    4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381153    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381156    4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381161    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381163    4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381166    4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381169    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381173    4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381175    4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:09.390437 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381178    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381181    4898 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381183    4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381186    4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381188    4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381192    4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381195    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381198    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381200    4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381203    4898 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381206    4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381208    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381211    4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381213    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381216    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381218    4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381221    4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381224    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381227    4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381229    4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:09.391008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381232    4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381235    4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381237    4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381240    4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381243    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381245    4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381249    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381252    4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381254    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381257    4898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381261    4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381264    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381267    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381269    4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381272    4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381274    4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381277    4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381280    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381282    4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381285    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:09.391842 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381288    4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381290    4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381294    4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381298    4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381301    4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381304    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381307    4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381310    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381312    4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381315    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381317    4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381320    4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381323    4898 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381325    4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381328    4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381330    4898 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381333    4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381335    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381340    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:09.392736 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381343    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381346    4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381350    4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381354    4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381357    4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381360    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381362    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381365    4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381368    4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381370    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381373    4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381375    4898 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381378    4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381381    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381383    4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381386    4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381388    4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381391    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381394    4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:09.393451 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381396    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381399    4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381401    4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381404    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381406    4898 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381409    4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381412    4898 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381414    4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381417    4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381419    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381422    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381424    4898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381430    4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.381455    4898 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381489    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381494    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:09.393960 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381497    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381502    4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381506    4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381510    4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381513    4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381516    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381519    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381522    4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381524    4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381527    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381530    4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381532    4898 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381535    4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381537    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381540    4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381543    4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381546    4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381548    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381551    4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:09.394377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381553    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381556    4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381558    4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381561    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381564    4898 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381566    4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381569    4898 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381571    4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381574    4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381578    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381581    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381583    4898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381586    4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381589    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381592    4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381595    4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381598    4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381601    4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381603    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381606    4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:09.395038 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381608    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381611    4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381614    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381616    4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381618    4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381621    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381623    4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381626    4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381629    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381631    4898 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381634    4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381636    4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381639    4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381642    4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381644    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381647    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381649    4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381652    4898 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381654    4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381657    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:09.395619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381659    4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381663    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381666    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381668    4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381671    4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381674    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381678    4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381680    4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381683    4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381686    4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381688    4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381692    4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381695    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381698    4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381700    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381703    4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381706    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381708    4898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381711    4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:09.396168 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381714    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381717    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381719    4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381722    4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381725    4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381727    4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381730    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381732    4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381735    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381738    4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381740    4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381744    4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381748    4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381751    4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381755    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381758    4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381761    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381763    4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381765    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:09.396656 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381771    4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381773    4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381776    4898 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381779    4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381781    4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381784    4898 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.381787    4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.385495    4898 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.385512    4898 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.385530    4898 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.385536    4898 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385593    4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385599    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385602    4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385605    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385608    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385611    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385614    4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385616    4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385619    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:09.397278 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385622    4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385624    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385627    4898 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385630    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385632    4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385635    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385638    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385641    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385644    4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385647    4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385649    4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385652    4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385654    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385657    4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385660    4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385662    4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385666    4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385670    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385673    4898 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:09.397773 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385676    4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385685    4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385688    4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385690    4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385693    4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385696    4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385698    4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385701    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385703    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385706    4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385708    4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385711    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385714    4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385716    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385719    4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385722    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385724    4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385727    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385730    4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385733    4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:09.398251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385735    4898 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385738    4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385741    4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385744    4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385747    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385750    4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385753    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385755    4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385758    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385760    4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385763    4898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385767    4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385771    4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385775    4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385786    4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385789    4898 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385791    4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385794    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385797    4898 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:09.398729 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385799    4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385802    4898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385805    4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385807    4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385810    4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385812    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385815    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385817    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385820    4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385822    4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385825    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385827    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385830    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385833    4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385835    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385838    4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385840    4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385843    4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385846    4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385848    4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:09.399217 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385851    4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385854    4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385856    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385859    4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385861    4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385864    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385867    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385869    4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385877    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385880    4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385883    4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385899    4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385904    4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385908    4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385912    4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385915    4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385919    4898 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385923    4898 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:09.399695 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.385927    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.385967    4898 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386228    4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386235    4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386240    4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386243    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386246    4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386249    4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386251    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386254    4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386257    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386260    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386263    4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386265    4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386268    4898 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:09.400143 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386270    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386273    4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386276    4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386278    4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386281    4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386284    4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386286    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386289    4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386297    4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386301    4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386303    4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386306    4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386308    4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386311    4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386313    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386316    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386318    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386321    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386324    4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:09.400547 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386326    4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386331    4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386334    4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386336    4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386339    4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386341    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386344    4898 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386346    4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386349    4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386352    4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386354    4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386357    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386359    4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386362    4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386365    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386367    4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386370    4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386372    4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386375    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:09.401023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386377    4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386380    4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386382    4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386390    4898 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386394    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386396    4898 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386399    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386401    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386404    4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386406    4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386409    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386411    4898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386414    4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386416    4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386419    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386422    4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386424    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386427    4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386430    4898 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386432    4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:09.401482 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386435    4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386437    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386441    4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386443    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386446    4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386454    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386457    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386460    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386462    4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386465    4898 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386467    4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386470    4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386472    4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386475    4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386477    4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386480    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386488    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386491    4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386493    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386496    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:09.402043 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386499    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386502    4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386504    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386507    4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386509    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386512    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386515    4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386517    4898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386520    4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386522    4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386525    4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386528    4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386530    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386533    4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386536    4898 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.386560    4898 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 07:24:09.402530 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386598    4898 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386602    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386605    4898 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386608    4898 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386611    4898 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386613    4898 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386616    4898 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386618    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386621    4898 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386623    4898 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386626    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386628    4898 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386630    4898 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386638    4898 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386641    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386644    4898 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386646    4898 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386649    4898 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386651    4898 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 07:24:09.402940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386654    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386656    4898 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386659    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386661    4898 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386664    4898 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386666    4898 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386669    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386671    4898 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386674    4898 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386676    4898 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386679    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386681    4898 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386684    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386686    4898 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386689    4898 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386692    4898 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386694    4898 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386697    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386699    4898 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386702    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 07:24:09.403409 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386704    4898 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386706    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386709    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386711    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386714    4898 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386716    4898 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386719    4898 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386728    4898 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386731    4898 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386733    4898 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386736    4898 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386738    4898 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386741    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386743    4898 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386746    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386748    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386751    4898 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386753    4898 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386756    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386758    4898 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 07:24:09.403908 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386761    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386763    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386766    4898 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386769    4898 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386772    4898 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386774    4898 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386776    4898 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386779    4898 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386781    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386784    4898 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386786    4898 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386791    4898 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386795    4898 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386799    4898 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386802    4898 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386805    4898 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386807    4898 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386810    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386813    4898 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 07:24:09.404436 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386815    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386824    4898 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386826    4898 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386829    4898 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386832    4898 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386834    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386837    4898 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386840    4898 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386842    4898 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386845    4898 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386848    4898 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386850    4898 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386853    4898 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386855    4898 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386858    4898 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386860    4898 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386863    4898 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386865    4898 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386868    4898 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 07:24:09.404914 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386870    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386873    4898 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386875    4898 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386878    4898 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386880    4898 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386884    4898 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386905    4898 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386909    4898 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:09.386912    4898 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.386973    4898 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.388006    4898 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.388050    4898 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.388338    4898 server.go:1014] "Starting client certificate rotation"
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.388493    4898 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.388585    4898 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 07:18:29 +0000 UTC" deadline="2028-04-02 05:44:35.625158488 +0000 UTC"
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.388651    4898 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12406h20m26.236509109s"
Nov 02 07:24:09.405401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.388908    4898 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 07:24:09.405833 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.390172    4898 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 07:24:09.405833 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.392776    4898 log.go:25] "Validated CRI v1 runtime API"
Nov 02 07:24:09.405833 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.394623    4898 log.go:25] "Validated CRI v1 image API"
Nov 02 07:24:09.405833 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.395470    4898 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 07:24:09.405833 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.397419    4898 fs.go:135] Filesystem UUIDs: map[13163f21-dcf2-446b-acc4-096087adf2ea:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c5bf0693-a6c6-4492-ae53-04d08eb06a4e:/dev/nvme0n1p3]
Nov 02 07:24:09.405999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.397445    4898 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c70c196a72bdf35fcd10843918f22e32bc3a6a546042d67faa1db8244cf97c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c70c196a72bdf35fcd10843918f22e32bc3a6a546042d67faa1db8244cf97c6/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d810d41c53333e445450a82ea59d891d3037c2302e589e0e914315efcaae687/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d810d41c53333e445450a82ea59d891d3037c2302e589e0e914315efcaae687/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/397c4955e3bebc429ee9892f20a9883bb475ad1f21c66f2b9433fb8f29f79bcc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/397c4955e3bebc429ee9892f20a9883bb475ad1f21c66f2b9433fb8f29f79bcc/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/500ef842aae7570570c53c39943b189bca7f232f753048216fd402c95d02ae62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/500ef842aae7570570c53c39943b189bca7f232f753048216fd402c95d02ae62/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/524a3a58f9688d283291a8da385130fa4eece4ee1b65a366f303f8fd22de50bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/524a3a58f9688d283291a8da385130fa4eece4ee1b65a366f303f8fd22de50bd/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bdcb3c6342c96f5b0fe38bb840d62ce06f3a49a0e7d2e03b2222a964528a197/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bdcb3c6342c96f5b0fe38bb840d62ce06f3a49a0e7d2e03b2222a964528a197/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d2612c2774448ed21ba7a86f46c5e7f2b47d9969c9a9f11654e50862790b3d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d2612c2774448ed21ba7a86f46c5e7f2b47d9969c9a9f11654e50862790b3d1/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69753ffa641658976099a31f4bba3482713ecdfaf02065de78a88924eb0c3633/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69753ffa641658976099a31f4bba3482713ecdfaf02065de78a88924eb0c3633/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7674727f9ddb2a9032353c555553c9d0c81f4dc11350933d54c9f03f9785637c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7674727f9ddb2a9032353c555553c9d0c81f4dc11350933d54c9f03f9785637c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f29cfc958f17f4eab80997a226f20717591bf7c9af2c6bda266dc72af7cda16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f29cfc958f17f4eab80997a226f20717591bf7c9af2c6bda266dc72af7cda16/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81925b6b568ae3709ef5db6166c59006ba56e66ad61d7776496b0abaf11e8cbf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81925b6b568ae3709ef5db6166c59006ba56e66ad61d7776496b0abaf11e8cbf/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d4309e6ba2b04806fc412a1d6481add54d7a87f5e976ce51cb9a883a0c08124/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d4309e6ba2b04806fc412a1d6481add54d7a87f5e976ce51cb9a883a0c08124/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/932b6142154dd73da54393785d8b89ba5d58709c497f2b30dc8573442ee1b10a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/932b6142154dd73da54393785d8b89ba5d58709c497f2b30dc8573442ee1b10a/userdata/shm major:0 minor:309 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acb44db5b2f90e52c9696bb3a11392dee0cdcd24b9635c8806a33259048792fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acb44db5b2f90e52c9696bb3a11392dee0cdcd24b9635c8806a33259048792fd/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b199a2a5e18e5bb1c841f6d0a38cee1d6f7e82f13c39bea903ce5d017916933b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b199a2a5e18e5bb1c841f6d0a38cee1d6f7e82f13c39bea903ce5d017916933b/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3ec61b2f9272c2ce53e71912f48ba76d08af83b6fc10424805a98bbef1f46e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3ec61b2f9272c2ce53e71912f48ba76d08af83b6fc10424805a98bbef1f46e3/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0403c57e79b789165972e9076d6419b76ed0168408c4779079dc501d6300472/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0403c57e79b789165972e9076d6419b76ed0168408c4779079dc501d6300472/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2b7759057ce265a011dcd82332b2fa9058a5b75714cd040beb4c48ae636e979/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2b7759057ce265a011dcd82332b2fa9058a5b75714cd040beb4c48ae636e979/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cae4447db8674668e0a1383b3e4b2dfdc808bf449b96c59514bea0bb2c9a8ac0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cae4447db8674668e0a1383b3e4b2dfdc808bf449b96c59514bea0bb2c9a8ac0/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dac0eec0e056dcff6da478c46b4c6500229124306772a1a808df1598c87e8a08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dac0eec0e056dcff6da478c46b4c6500229124306772a1a808df1598c87e8a08/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3d627c6e8421ae277ab8773485b04ecbef13520834f978ec6dcc4191da38a1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3d627c6e8421ae277ab8773485b04ecbef13520834f978ec6dcc4191da38a1a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6e2094399a94328715f9a2eff4342789fb49efbd39e8f6f40a4841d1f1c3f55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6e2094399a94328715f9a2eff4342789fb49efbd39e8f6f40a4841d1f1c3f55/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee36e83bc2f2a81f701cf01449a70372e834c9d6115cc971a2193f2ffb154bac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee36e83bc2f2a81f701cf01449a70372e834c9d6115cc971a2193f2ffb154bac/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0059120b-d9ca-462c-a74b-bb99ea44c3d7/volumes/kubernetes.io~projected/kube-api-access-gxct4:{mountpoint:/var/lib/kubelet/pods/0059120b-d9ca-462c-a74b-bb99ea44c3d7/volumes/kubernetes.io~projected/kube-api-access-gxct4 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0196d04c-bd2b-48c7-bdb9-d31d83753247/volumes/kubernetes.io~projected/kube-api-access-d4pcq:{mountpoint:/var/lib/kubelet/pods/0196d04c-bd2b-48c7-bdb9-d31d83753247/volumes/kubernetes.io~projected/kube-api-access-d4pcq major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0196d04c-bd2b-48c7-bdb9-d31d83753247/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0196d04c-bd2b-48c7-bdb9-d31d83753247/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400/volumes/kubernetes.io~projected/kube-api-access-mxkxw:{mountpoint:/var/lib/kubelet/pods/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400/volumes/kubernetes.io~projected/kube-api-access-mxkxw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08bc1af6-1c3b-4740-9dc0-fa309e83b78a/volumes/kubernetes.io~projected/kube-api-access-bm5vh:{mountpoint:/var/lib/kubelet/pods/08bc1af6-1c3b-4740-9dc0-fa309e83b78a/volumes/kubernetes.io~projected/kube-api-access-bm5vh major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes/kubernetes.io~projected/kube-api-access-rdn4r:{mountpoint:/var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes/kubernetes.io~projected/kube-api-access-rdn4r major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0/volumes/kubernetes.io~projected/kube-api-access-v8jc5:{mountpoint:/var/lib/kubelet/pods/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0/volumes/kubernetes.io~projected/kube-api-access-v8jc5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/437c4525-b041-4931-ba40-99c518193657/volumes/kubernetes.io~projected/kube-api-access-ldhmb:{mountpoint:/var/lib/kubelet/pods/437c4525-b041-4931-ba40-99c518193657/volumes/kubernetes.io~projected/kube-api-access-ldhmb major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/531c943e-bc09-4336-9d48-065b8b716dc4/volumes/kubernetes.io~projected/kube-api-access-vbfdt:{mountpoint:/var/lib/kubelet/pods/531c943e-bc09-4336-9d48-065b8b716dc4/volumes/kubernetes.io~projected/kube-api-access-vbfdt major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/531c943e-bc09-4336-9d48-065b8b716dc4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/531c943e-bc09-4336-9d48-065b8b716dc4/volumes/kubernetes.io~secret/serving-cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/539550af-7526-447a-b6e3-d268f758aeb1/volumes/kubernetes.io~projected/kube-api-access-d8qvr:{mountpoint:/var/lib/kubelet/pods/539550af-7526-447a-b6e3-d268f758aeb1/volumes/kubernetes.io~projected/kube-api-access-d8qvr major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54a48a78-5f18-4e96-be7b-36f356cc9b39/volumes/kubernetes.io~projected/kube-api-access-k2qj8:{mountpoint:/var/lib/kubelet/pods/54a48a78-5f18-4e96-be7b-36f356cc9b39/volumes/kubernetes.io~projected/kube-api-access-k2qj8 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54a48a78-5f18-4e96-be7b-36f356cc9b39/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/54a48a78-5f18-4e96-be7b-36f356cc9b39/volumes/kubernetes.io~secret/cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c54280e-7b8b-4f95-bbe6-a93dd82018eb/volumes/kubernetes.io~projected/kube-api-access-9sd8w:{mountpoint:/var/lib/kubelet/pods/6c54280e-7b8b-4f95-bbe6-a93dd82018eb/volumes/kubernetes.io~projected/kube-api-access-9sd8w major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75d5ea8e-8d3d-4113-926c-5b29267d479b/volumes/kubernetes.io~projected/kube-api-access-6jrmw:{mountpoint:/var/lib/kubelet/pods/75d5ea8e-8d3d-4113-926c-5b29267d479b/volumes/kubernetes.io~projected/kube-api-access-6jrmw major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7886c1fa-91a5-4ac1-9c59-98a8268bb05e/volumes/kubernetes.io~projected/kube-api-access-9fs4t:{mountpoint:/var/lib/kubelet/pods/7886c1fa-91a5-4ac1-9c59-98a8268bb05e/volumes/kubernetes.io~projected/kube-api-access-9fs4t major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78f8157e-9211-4f25-886e-4a9e9338e730/volumes/kubernetes.io~projected/kube-api-access-mg6jk:{mountpoint:/var/lib/kubelet/pods/78f8157e-9211-4f25-886e-4a9e9338e730/volumes/kubernetes.io~projected/kube-api-access-mg6jk major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bf60630-1d98-4928-a2bf-dda7a1e1a110/volumes/kubernetes.io~projected/kube-api-access-rx642:{mountpoint:/var/lib/kubelet/pods/7bf60630-1d98-4928-a2bf-dda7a1e1a110/volumes/kubernetes.io~projected/kube-api-access-rx642 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e97eaa5-7df2-484e-a200-95f3fd1114d9/volumes/kubernetes.io~projected/kube-api-access-m2bbr:{mountpoint:/var/lib/kubelet/pods/7e97eaa5-7df2-484e-a200-95f3fd1114d9/volumes/kubernetes.io~projected/kube-api-access-m2bbr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2/volumes/kubernetes.io~projected/kube-api-access-j5bj8:{mountpoint:/var/lib/kubelet/pods/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2/volumes/kubernetes.io~projected/kube-api-access-j5bj8 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/948eb959-3a16-4ea4-a87b-9497f8bf46d9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/948eb959-3a16-4ea4-a87b-9497f8bf46d9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/948eb959-3a16-4ea4-a87b-9497f8bf46d9/volumes/kubernetes.io~projected/kube-api-access-zprsb:{mountpoint:/var/lib/kubelet/pods/948eb959-3a16-4ea4-a87b-9497f8bf46d9/volumes/kubernetes.io~projected/kube-api-access-zprsb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/948eb959-3a16-4ea4-a87b-9497f8bf46d9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/948eb959-3a16-4ea4-a87b-9497f8bf46d9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76/volumes/kubernetes.io~projected/kube-api-access-jl4ww:{mountpoint:/var/lib/kubelet/pods/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76/volumes/kubernetes.io~projected/kube-api-access-jl4ww major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bb5657a-bb11-4279-86f6-35118656d990/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9bb5657a-bb11-4279-86f6-35118656d990/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes/kubernetes.io~projected/kube-api-access-9cqp8:{mountpoint:/var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes/kubernetes.io~projected/kube-api-access-9cqp8 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6da9cb8-1262-4288-a7fb-5991fff5c41a/volumes/kubernetes.io~projected/kube-api-access-tfqsf:{mountpoint:/var/lib/kubelet/pods/a6da9cb8-1262-4288-a7fb-5991fff5c41a/volumes/kubernetes.io~projected/kube-api-access-tfqsf major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b93e0f22-01e7-4128-a999-da9b96e029f8/volumes/kubernetes.io~projected/kube-api-access-qqpcg:{mountpoint:/var/lib/kubelet/pods/b93e0f22-01e7-4128-a999-da9b96e029f8/volumes/kubernetes.io~projected/kube-api-access-qqpcg major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b93e0f22-01e7-4128-a999-da9b96e029f8/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/b93e0f22-01e7-4128-a999-da9b96e029f8/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b93e0f22-01e7-4128-a999-da9b96e029f8/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/b93e0f22-01e7-4128-a999-da9b96e029f8/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb81587c-bf28-4f77-ac53-02cc794c5365/volumes/kubernetes.io~projected/kube-api-access-spj5p:{mountpoint:/var/lib/kubelet/pods/bb81587c-bf28-4f77-ac53-02cc794c5365/volumes/kubernetes.io~projected/kube-api-access-spj5p major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c90ce63c-e5e9-4edf-8a1c-78ee6d092633/volumes/kubernetes.io~projected/kube-api-access-crjhf:{mountpoint:/var/lib/kubelet/pods/c90ce63c-e5e9-4edf-8a1c-78ee6d092633/volumes/kubernetes.io~projected/kube-api-access-crjhf major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d338565e-a9f0-44e8-ba6c-e0d88c253a10/volumes/kubernetes.io~projected/kube-api-access-7fwtm:{mountpoint:/var/lib/kubelet/pods/d338565e-a9f0-44e8-ba6c-e0d88c253a10/volumes/kubernetes.io~projected/kube-api-access-7fwtm major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c/volumes/kubernetes.io~projected/kube-api-access-qh46m:{mountpoint:/var/lib/kubelet/pods/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c/volumes/kubernetes.io~projected/kube-api-access-qh46m major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4ba5ec9-5277-460a-976e-3b1f154f7623/volumes/kubernetes.io~projected/kube-api-access-dbdl7:{mountpoint:/var/lib/kubelet/pods/f4ba5ec9-5277-460a-976e-3b1f154f7623/volumes/kubernetes.io~projected/kube-api-access-dbdl7 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4ba5ec9-5277-460a-976e-3b1f154f7623/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f4ba5ec9-5277-460a-976e-3b1f154f7623/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f85f18d7-a8b8-4634-af97-eb4e71dc5000/volumes/kubernetes.io~projected/kube-api-access-rqvfj:{mountpoint:/var/lib/kubelet/pods/f85f18d7-a8b8-4634-af97-eb4e71dc5000/volumes/kubernetes.io~projected/kube-api-access-rqvfj major:0 minor:207 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/13d8079aa158b50dd72b467ae0a42fa05fa4ab1782762c23ab3eafe0ed57761d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d9568b6ad4b023e29870735473474ec746c13f5953ae5bd40ca884d3dbe187a2/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f61dee64969bca63ab7bcbdbf318450aa49fa7211e88e3f83dbd544f050de8e8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/cd556d0ae19c6e99a88e5770007749974efea93a97fffd53fffc6e61ba7fa6c1/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/cd2a341e1259bc7302518f8f72360ffff474c0d2610bb0be1ee53dbec71e68c6/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b4bf3ee80b44566dfac1e65a3bdf6599ab985005266e80d283343312efcee5f0/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a2e308f2bf317e938536cb755c2bd47177a40d56282dce72d7ad58c4ebe23506/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/da90248375de803da10d9a23b322da321739ad7b59982a17e18751771d3657ad/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/0340cb92e36139de02bfdee2e43750243e66fa40267400e82c7ca15130b28351/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/38e5c3da1ec4005fb8434cea43d47cd22d39acea76d416850d94d4ab464a0dfc/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/3dc32ffdf67759d03d79276264e70d394f6b6be68e68c72f088949a96157b77b/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/ac8c7fc8942afd9833e479d7723931a37ca26cae8052cc502d63f713f52c2c45/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/99e297ceb7c9b938d84bd23931565fbec42e68f8067dfb5fd1b760e5e1bf7040/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1876207f197c24ede32b0f800f7e5973185d48436f43f7b4cb29fef473be6cce/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/17e7935fbc8d716b23469129b8e63d90da5aa44002a4889e95ea0d6ddbdafad7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/249660c221c2f42d7968e31ee36301767d123e1aa60da2f88b5b98bb69c6747d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/15162413a76f822065d706a100d26a2fd4f926bb76e326b4ad1d45e7162f39fa/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ac3064615910a6aa36fb935d4cbee4399252aba3491d71cdafcf768cbdc07ca9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/cdb2b0c9de4951a77d96de1c61d221172f1b220b69cc6c3e6731b9da010bd2da/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/970d2a8e9b0bc92143a4739648664ca38df0c3a8e93ba77abfac3cc1305ebba6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/51b4e2dc3c396a9cf694a8d756f32b0ad028180e9d18f15550e3355e7497aaad/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/f56642970bb0931f870a9ba5853116e27dd81dd0da8fb20139fcbcf12d7e147b/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/10480435d11b90421c61bee4fb62b2c91da3f83668eb3146455d17e930c65c42/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/1a9d09730f41421bc2a6b005985795f32a87bc05192c27ca0e265a17d9b88062/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/2cbd4737f58987588a2c4275a7289391dd275b44ad615463dc762e7040d64452/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/b52d54586496234d88b2d791526ca8af5df0e8fea9877b146c41451b4f3ec72b/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/74407961039a022cbdab9cbdb9bcdf3cfa9474238e3921441586899acfbe240c/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/c5ea407fe44ce063f9860b7a13cec8f558d280b5120bf68b453e70dd87db24bc/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/6ad95093878a14c98506b9454ab75656723cd48ce94383c956150920fae81d0f/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/eccf86fcbdb3b2b431edf3f4cff8871348516450d04822284fd0b594b22ac86e/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/bbc51391ffb9781bea630a065ce025f64a6d4b2e18ca9fcb7d12f1206404cb8c/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/84f875793fe78f86e090f6890407c77c6354371c681db3f0abfdbd697296c1a8/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/2d7aa88003293bc387f46c2e0fd3776fa2ada05859440ca6dbf3be877161bc77/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/2e60b7dcbb1d29c31f3239c6196cb61d9c80748e2e711923f709349acb77554d/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/21d34a7cdadf06d27f4b980a7c6191351c41dc4aebda089a6d3e475eaece02a4/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/27172be853fe28c0db327509b0968dd1ee7155ad4c7bbc38689cf1ad1907b64e/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/175a92454aaef41f7fb4f7147559f3c3895f3d37d56b00de47b67f9d6f086c0c/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/1abb1bda7dabd38d1bb317ddbad24e4fa8486b4414c2bd5f2597fb943085053e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/dbd059014f3e61fb7926b5817e3705638085e98fc1eda98a383ad075b23ae1c7/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/f78d8c25fa6a62dc8cecb9e4129d1dd02354c8d8e9ffd1accfc920e7c642e422/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/7a65b5f5e63beba3fbb467bdb7d9f3650d2e9bf90139d141f7c450b04849ce4d/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/d9466110cf53920565c21439fb574298f794b54f166b91b5c3e6bc5287521209/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/5e69d48cab6c2aeea688a5832fd7894aecab96d83a14d24f968c7ff039199320/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/3e89e7b16101ee73ff1f6fd9f64ffae4a0aa629eddbb85c031c22e08d27f20b3/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/f83cea5252a6ea62919487e4181879780b75aaaf5531983c65c8753948a1591d/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/08003c486f9d4ea6067da2cbb3e38dce50501d6fbff523993c35e6eaf5381bab/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/08eddfc02b7a17bb59665fce28a96d4819bb881acab4048a23c0e8cc03001cfe/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/74c1fb543f98550f6645a02797dfc9cf520269663bd285b1fbf6fd616c7e607d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ec49e130b8d34e0aa5c2a1e7606ffe5d930d758fd497897992ea94dc09294d37/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/c49068c70ba6c489369ff03b278b4b4a03335a451cbd726a5c63957527cd8101/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/d60da74b052c38f1641e8c31fba9930f504b3bfe6720e12fcc9ae4a0c5d421c9/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c7d9328fd0c6d0ef6b45c30a5dfac196ecea1a70c4cb21e6967cab725998ee51/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/b1f13047d1396d9d8a645a08442f7d0ebc9783434de9c99e16907a2202cad0d9/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/4da138c05b7094c475b38c92af0ff5a8fabdf72e23738a817ad39cbc3ae0d00d/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/281f964981d61d4a6a25d5b8548f035e2bf72a3f611b88b44015deafc4c8cfb2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d632ce6d9e62e4592ec05a6eac10ec8381e47468e3de51c8fa9341c629efa7f9/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 07:24:09.406218 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.405374    4898 manager.go:217] Machine: {Timestamp:2026-11-02 07:24:09.404857792 +0000 UTC m=+0.095184746 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116064 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b7a059e8f80beaee5e77160f981d6 SystemUUID:ec2b7a05-9e8f-80be-aee5-e77160f981d6 BootID:81f6a676-8870-443f-96a5-912e855b335d Filesystems:[{Device:/var/lib/kubelet/pods/6c54280e-7b8b-4f95-bbe6-a93dd82018eb/volumes/kubernetes.io~projected/kube-api-access-9sd8w DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/948eb959-3a16-4ea4-a87b-9497f8bf46d9/volumes/kubernetes.io~projected/kube-api-access-zprsb DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f29cfc958f17f4eab80997a226f20717591bf7c9af2c6bda266dc72af7cda16/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f85f18d7-a8b8-4634-af97-eb4e71dc5000/volumes/kubernetes.io~projected/kube-api-access-rqvfj DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acb44db5b2f90e52c9696bb3a11392dee0cdcd24b9635c8806a33259048792fd/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6da9cb8-1262-4288-a7fb-5991fff5c41a/volumes/kubernetes.io~projected/kube-api-access-tfqsf DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3ec61b2f9272c2ce53e71912f48ba76d08af83b6fc10424805a98bbef1f46e3/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4ba5ec9-5277-460a-976e-3b1f154f7623/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b93e0f22-01e7-4128-a999-da9b96e029f8/volumes/kubernetes.io~projected/kube-api-access-qqpcg DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54a48a78-5f18-4e96-be7b-36f356cc9b39/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/500ef842aae7570570c53c39943b189bca7f232f753048216fd402c95d02ae62/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0059120b-d9ca-462c-a74b-bb99ea44c3d7/volumes/kubernetes.io~projected/kube-api-access-gxct4 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0403c57e79b789165972e9076d6419b76ed0168408c4779079dc501d6300472/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee36e83bc2f2a81f701cf01449a70372e834c9d6115cc971a2193f2ffb154bac/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7886c1fa-91a5-4ac1-9c59-98a8268bb05e/volumes/kubernetes.io~projected/kube-api-access-9fs4t DeviceMajor:0 DeviceMinor:331 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes/kubernetes.io~projected/kube-api-access-9cqp8 DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/437c4525-b041-4931-ba40-99c518193657/volumes/kubernetes.io~projected/kube-api-access-ldhmb DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d338565e-a9f0-44e8-ba6c-e0d88c253a10/volumes/kubernetes.io~projected/kube-api-access-7fwtm DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/524a3a58f9688d283291a8da385130fa4eece4ee1b65a366f303f8fd22de50bd/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c70c196a72bdf35fcd10843918f22e32bc3a6a546042d67faa1db8244cf97c6/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/7bf60630-1d98-4928-a2bf-dda7a1e1a110/volumes/kubernetes.io~projected/kube-api-access-rx642 DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76/volumes/kubernetes.io~projected/kube-api-access-jl4ww DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7674727f9ddb2a9032353c555553c9d0c81f4dc11350933d54c9f03f9785637c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/539550af-7526-447a-b6e3-d268f758aeb1/volumes/kubernetes.io~projected/kube-api-access-d8qvr DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78f8157e-9211-4f25-886e-4a9e9338e730/volumes/kubernetes.io~projected/kube-api-access-mg6jk DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dac0eec0e056dcff6da478c46b4c6500229124306772a1a808df1598c87e8a08/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb81587c-bf28-4f77-ac53-02cc794c5365/volumes/kubernetes.io~projected/kube-api-access-spj5p DeviceMajor:0 DeviceMinor:330 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cae4447db8674668e0a1383b3e4b2dfdc808bf449b96c59514bea0bb2c9a8ac0/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69753ffa641658976099a31f4bba3482713ecdfaf02065de78a88924eb0c3633/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9bb5657a-bb11-4279-86f6-35118656d990/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:274 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/932b6142154dd73da54393785d8b89ba5d58709c497f2b30dc8573442ee1b10a/userdata/shm DeviceMajor:0 DeviceMinor:309 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/948eb959-3a16-4ea4-a87b-9497f8bf46d9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4ba5ec9-5277-460a-976e-3b1f154f7623/volumes/kubernetes.io~projected/kube-api-access-dbdl7 DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81925b6b568ae3709ef5db6166c59006ba56e66ad61d7776496b0abaf11e8cbf/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d810d41c53333e445450a82ea59d891d3037c2302e589e0e914315efcaae687/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b93e0f22-01e7-4128-a999-da9b96e029f8/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08bc1af6-1c3b-4740-9dc0-fa309e83b78a/volumes/kubernetes.io~projected/kube-api-access-bm5vh DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2/volumes/kubernetes.io~projected/kube-api-access-j5bj8 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6e2094399a94328715f9a2eff4342789fb49efbd39e8f6f40a4841d1f1c3f55/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400/volumes/kubernetes.io~projected/kube-api-access-mxkxw DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0/volumes/kubernetes.io~projected/kube-api-access-v8jc5 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2b7759057ce265a011dcd82332b2fa9058a5b75714cd040beb4c48ae636e979/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0196d04c-bd2b-48c7-bdb9-d31d83753247/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0196d04c-bd2b-48c7-bdb9-d31d83753247/volumes/kubernetes.io~projected/kube-api-access-d4pcq DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54a48a78-5f18-4e96-be7b-36f356cc9b39/volumes/kubernetes.io~projected/kube-api-access-k2qj8 DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c90ce63c-e5e9-4edf-8a1c-78ee6d092633/volumes/kubernetes.io~projected/kube-api-access-crjhf DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3d627c6e8421ae277ab8773485b04ecbef13520834f978ec6dcc4191da38a1a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes/kubernetes.io~projected/kube-api-access-rdn4r DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b199a2a5e18e5bb1c841f6d0a38cee1d6f7e82f13c39bea903ce5d017916933b/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bdcb3c6342c96f5b0fe38bb840d62ce06f3a49a0e7d2e03b2222a964528a197/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b93e0f22-01e7-4128-a999-da9b96e029f8/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/397c4955e3bebc429ee9892f20a9883bb475ad1f21c66f2b9433fb8f29f79bcc/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c/volumes/kubernetes.io~projected/kube-api-access-qh46m DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d2612c2774448ed21ba7a86f46c5e7f2b47d9969c9a9f11654e50862790b3d1/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/948eb959-3a16-4ea4-a87b-9497f8bf46d9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/75d5ea8e-8d3d-4113-926c-5b29267d479b/volumes/kubernetes.io~projected/kube-api-access-6jrmw DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/531c943e-bc09-4336-9d48-065b8b716dc4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e97eaa5-7df2-484e-a200-95f3fd1114d9/volumes/kubernetes.io~projected/kube-api-access-m2bbr DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d4309e6ba2b04806fc412a1d6481add54d7a87f5e976ce51cb9a883a0c08124/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/531c943e-bc09-4336-9d48-065b8b716dc4/volumes/kubernetes.io~projected/kube-api-access-vbfdt DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0c70c196a72bdf3 MacAddress:f6:58:61:e7:b0:c4 Speed:10000 Mtu:8901} {Name:397c4955e3bebc4 MacAddress:f2:4c:d6:c9:91:96 Speed:10000 Mtu:8901} {Name:500ef842aae7570 MacAddress:0a:0e:f0:d6:20:e7 Speed:10000 Mtu:8901} {Name:524a3a58f9688d2 MacAddress:46:54:34:4c:60:d5 Speed:10000 Mtu:8901} {Name:5d2612c2774448e MacAddress:56:0a:5e:6d:c4:c3 Speed:10000 Mtu:8901} {Name:69753ffa6416589 MacAddress:02:1d:6c:45:0d:be Speed:10000 Mtu:8901} {Name:8d4309e6ba2b048 MacAddress:ce:da:33:a8:86:4e Speed:10000 Mtu:8901} {Name:932b6142154dd73 MacAddress:b6:5d:80:7e:79:8b Speed:10000 Mtu:8901} {Name:acb44db5b2f90e5 MacAddress:52:19:63:68:0f:97 Speed:10000 Mtu:8901} {Name:b199a2a5e18e5bb MacAddress:2a:2b:df:60:5c:33 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d9:19:52:87 Speed:0 Mtu:9001} {Name:br-int MacAddress:ca:bc:86:74:a7:45 Speed:0 Mtu:8901} {Name:dac0eec0e056dcf MacAddress:a6:42:e8:09:37:b9 Speed:10000 Mtu:8901} {Name:e6e2094399a9432 MacAddress:3a:12:84:96:ea:67 Speed:10000 Mtu:8901} {Name:ee36e83bc2f2a81 MacAddress:ca:4e:88:85:bb:6f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d9:19:52:87 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b2:47:9f:46:1c:c6 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ee:8f:b5:e8:ba:f4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 07:24:09.406218 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.405756    4898 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 07:24:09.406218 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.405802    4898 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 07:24:09.406218 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406175    4898 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 07:24:09.406459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406197    4898 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-79.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 07:24:09.406459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406332    4898 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 07:24:09.406459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406339    4898 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 07:24:09.406459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406352    4898 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 07:24:09.406459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406365    4898 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 07:24:09.406459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406441    4898 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 07:24:09.406621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406610    4898 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 07:24:09.406739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406729    4898 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 07:24:09.406780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406770    4898 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 07:24:09.406811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406784    4898 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 07:24:09.406811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406794    4898 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 07:24:09.406811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406803    4898 kubelet.go:405] "Adding apiserver pod source"
Nov 02 07:24:09.406811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.406811    4898 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 07:24:09.407941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.407922    4898 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 07:24:09.407941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.407940    4898 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 07:24:09.408441 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.408421    4898 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 07:24:09.411787 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.411769    4898 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 07:24:09.411787 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.411785    4898 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 07:24:09.411930 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.411882    4898 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 07:24:09.412095 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412084    4898 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 07:24:09.412129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412113    4898 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 07:24:09.412217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412207    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 07:24:09.412253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412222    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 07:24:09.412253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412228    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 07:24:09.412253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412233    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 07:24:09.412253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412238    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 07:24:09.412253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412244    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 07:24:09.412398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412268    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 07:24:09.412398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412275    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 07:24:09.412398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412282    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 07:24:09.412398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412288    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 07:24:09.412398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412296    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 07:24:09.412398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412305    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 07:24:09.412398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412328    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 07:24:09.412398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.412333    4898 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 07:24:09.415674 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.415658    4898 server.go:1263] "Started kubelet"
Nov 02 07:24:09.415751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.415718    4898 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 07:24:09.417059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.416091    4898 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 07:24:09.417059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.416196    4898 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 07:24:09.417059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.416484    4898 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 07:24:09.417059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.416935    4898 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 07:24:09.416228 ip-10-0-134-79 systemd[1]: Started Kubernetes Kubelet.
Nov 02 07:24:09.419395 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.419376    4898 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 07:24:09.419712 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.419600    4898 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 07:24:09.419712 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.419622    4898 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 07:24:09.421034 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:09.421003    4898 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-79.ec2.internal\" not found"
Nov 02 07:24:09.421130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.419607    4898 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 07:24:09.421130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.421060    4898 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 07:24:09.422388 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.422345    4898 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 07:18:31 +0000 UTC" deadline="2028-05-22 10:57:40.976646207 +0000 UTC"
Nov 02 07:24:09.422388 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.422387    4898 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13611h33m31.554262608s"
Nov 02 07:24:09.423033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.423003    4898 factory.go:55] Registering systemd factory
Nov 02 07:24:09.423033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.423035    4898 factory.go:223] Registration of the systemd container factory successfully
Nov 02 07:24:09.423699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.423666    4898 factory.go:153] Registering CRI-O factory
Nov 02 07:24:09.423768 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.423701    4898 factory.go:223] Registration of the crio container factory successfully
Nov 02 07:24:09.423801 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.423771    4898 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 07:24:09.423801 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.423789    4898 factory.go:103] Registering Raw factory
Nov 02 07:24:09.423861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.423804    4898 manager.go:1199] Started watching for new ooms in manager
Nov 02 07:24:09.424184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.424172    4898 manager.go:319] Starting recovery of all containers
Nov 02 07:24:09.425966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.425920    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="539550af-7526-447a-b6e3-d268f758aeb1" volumeName="kubernetes.io/projected/539550af-7526-447a-b6e3-d268f758aeb1-kube-api-access-d8qvr" seLinuxMountContext=""
Nov 02 07:24:09.426061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.425973    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c54280e-7b8b-4f95-bbe6-a93dd82018eb" volumeName="kubernetes.io/configmap/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-config" seLinuxMountContext=""
Nov 02 07:24:09.426061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.425989    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75d5ea8e-8d3d-4113-926c-5b29267d479b" volumeName="kubernetes.io/configmap/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-daemon-config" seLinuxMountContext=""
Nov 02 07:24:09.426061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426001    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af97e4d-867f-4a6c-8eb3-43d3ef4dce76" volumeName="kubernetes.io/projected/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-kube-api-access-jl4ww" seLinuxMountContext=""
Nov 02 07:24:09.426061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426019    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ff6ca66-e0d8-441f-8822-022929c12120" volumeName="kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-kube-api-access-9cqp8" seLinuxMountContext=""
Nov 02 07:24:09.426061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426032    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0196d04c-bd2b-48c7-bdb9-d31d83753247" volumeName="kubernetes.io/configmap/0196d04c-bd2b-48c7-bdb9-d31d83753247-config" seLinuxMountContext=""
Nov 02 07:24:09.426061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426048    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08bc1af6-1c3b-4740-9dc0-fa309e83b78a" volumeName="kubernetes.io/configmap/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-config-volume" seLinuxMountContext=""
Nov 02 07:24:09.426061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426060    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08bc1af6-1c3b-4740-9dc0-fa309e83b78a" volumeName="kubernetes.io/projected/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-kube-api-access-bm5vh" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426081    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="124a32e6-1529-4b84-aa39-dda6f9332364" volumeName="kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-installation-pull-secrets" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426094    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54a48a78-5f18-4e96-be7b-36f356cc9b39" volumeName="kubernetes.io/secret/54a48a78-5f18-4e96-be7b-36f356cc9b39-cert" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426109    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75d5ea8e-8d3d-4113-926c-5b29267d479b" volumeName="kubernetes.io/projected/75d5ea8e-8d3d-4113-926c-5b29267d479b-kube-api-access-6jrmw" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426140    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af97e4d-867f-4a6c-8eb3-43d3ef4dce76" volumeName="kubernetes.io/secret/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-serving-cert" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426154    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4ba5ec9-5277-460a-976e-3b1f154f7623" volumeName="kubernetes.io/configmap/f4ba5ec9-5277-460a-976e-3b1f154f7623-config" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426170    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ff6ca66-e0d8-441f-8822-022929c12120" volumeName="kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-trusted-ca" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426187    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ff6ca66-e0d8-441f-8822-022929c12120" volumeName="kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426205    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4ba5ec9-5277-460a-976e-3b1f154f7623" volumeName="kubernetes.io/projected/f4ba5ec9-5277-460a-976e-3b1f154f7623-kube-api-access-dbdl7" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426218    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3" volumeName="kubernetes.io/secret/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3-agent-certs" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426232    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="531c943e-bc09-4336-9d48-065b8b716dc4" volumeName="kubernetes.io/secret/531c943e-bc09-4336-9d48-065b8b716dc4-serving-cert" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426249    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="948eb959-3a16-4ea4-a87b-9497f8bf46d9" volumeName="kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-env-overrides" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426261    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ff6ca66-e0d8-441f-8822-022929c12120" volumeName="kubernetes.io/empty-dir/9ff6ca66-e0d8-441f-8822-022929c12120-ca-trust-extracted" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426278    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d338565e-a9f0-44e8-ba6c-e0d88c253a10" volumeName="kubernetes.io/projected/d338565e-a9f0-44e8-ba6c-e0d88c253a10-kube-api-access-7fwtm" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426291    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0059120b-d9ca-462c-a74b-bb99ea44c3d7" volumeName="kubernetes.io/projected/0059120b-d9ca-462c-a74b-bb99ea44c3d7-kube-api-access-gxct4" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426305    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="539550af-7526-447a-b6e3-d268f758aeb1" volumeName="kubernetes.io/empty-dir/539550af-7526-447a-b6e3-d268f758aeb1-tmp-dir" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426321    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="948eb959-3a16-4ea4-a87b-9497f8bf46d9" volumeName="kubernetes.io/secret/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426334    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="124a32e6-1529-4b84-aa39-dda6f9332364" volumeName="kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-trusted-ca" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426351    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a97cd1c-a0a6-4ea6-95de-ffadd05592b2" volumeName="kubernetes.io/empty-dir/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-tmp" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426365    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="948eb959-3a16-4ea4-a87b-9497f8bf46d9" volumeName="kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426378    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4ba5ec9-5277-460a-976e-3b1f154f7623" volumeName="kubernetes.io/configmap/f4ba5ec9-5277-460a-976e-3b1f154f7623-trusted-ca" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426405    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e97eaa5-7df2-484e-a200-95f3fd1114d9" volumeName="kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cni-binary-copy" seLinuxMountContext=""
Nov 02 07:24:09.426414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426422    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e97eaa5-7df2-484e-a200-95f3fd1114d9" volumeName="kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426434    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af97e4d-867f-4a6c-8eb3-43d3ef4dce76" volumeName="kubernetes.io/configmap/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-service-ca-bundle" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426449    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c90ce63c-e5e9-4edf-8a1c-78ee6d092633" volumeName="kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426466    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="124a32e6-1529-4b84-aa39-dda6f9332364" volumeName="kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-registry-certificates" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426481    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="437c4525-b041-4931-ba40-99c518193657" volumeName="kubernetes.io/projected/437c4525-b041-4931-ba40-99c518193657-kube-api-access-ldhmb" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426501    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3" volumeName="kubernetes.io/configmap/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3-konnectivity-ca" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426516    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a97cd1c-a0a6-4ea6-95de-ffadd05592b2" volumeName="kubernetes.io/projected/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-kube-api-access-j5bj8" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426532    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af97e4d-867f-4a6c-8eb3-43d3ef4dce76" volumeName="kubernetes.io/empty-dir/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-snapshots" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426546    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ff6ca66-e0d8-441f-8822-022929c12120" volumeName="kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-bound-sa-token" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426559    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b93e0f22-01e7-4128-a999-da9b96e029f8" volumeName="kubernetes.io/projected/b93e0f22-01e7-4128-a999-da9b96e029f8-kube-api-access-qqpcg" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426575    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0196d04c-bd2b-48c7-bdb9-d31d83753247" volumeName="kubernetes.io/secret/0196d04c-bd2b-48c7-bdb9-d31d83753247-serving-cert" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426593    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c54280e-7b8b-4f95-bbe6-a93dd82018eb" volumeName="kubernetes.io/projected/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-kube-api-access-9sd8w" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426610    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bf60630-1d98-4928-a2bf-dda7a1e1a110" volumeName="kubernetes.io/projected/7bf60630-1d98-4928-a2bf-dda7a1e1a110-kube-api-access-rx642" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426622    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e97eaa5-7df2-484e-a200-95f3fd1114d9" volumeName="kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426635    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a97cd1c-a0a6-4ea6-95de-ffadd05592b2" volumeName="kubernetes.io/empty-dir/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-tuned" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426652    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af97e4d-867f-4a6c-8eb3-43d3ef4dce76" volumeName="kubernetes.io/empty-dir/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-tmp" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426665    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="531c943e-bc09-4336-9d48-065b8b716dc4" volumeName="kubernetes.io/projected/531c943e-bc09-4336-9d48-065b8b716dc4-kube-api-access-vbfdt" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426684    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="948eb959-3a16-4ea4-a87b-9497f8bf46d9" volumeName="kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovnkube-config" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426698    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400" volumeName="kubernetes.io/configmap/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-iptables-alerter-script" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426711    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400" volumeName="kubernetes.io/projected/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-kube-api-access-mxkxw" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426731    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="124a32e6-1529-4b84-aa39-dda6f9332364" volumeName="kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-bound-sa-token" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426744    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7886c1fa-91a5-4ac1-9c59-98a8268bb05e" volumeName="kubernetes.io/projected/7886c1fa-91a5-4ac1-9c59-98a8268bb05e-kube-api-access-9fs4t" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426761    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78f8157e-9211-4f25-886e-4a9e9338e730" volumeName="kubernetes.io/projected/78f8157e-9211-4f25-886e-4a9e9338e730-kube-api-access-mg6jk" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426775    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ff6ca66-e0d8-441f-8822-022929c12120" volumeName="kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-installation-pull-secrets" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426800    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06de30c9-de01-4f63-859a-23df97972617" volumeName="kubernetes.io/configmap/06de30c9-de01-4f63-859a-23df97972617-nginx-conf" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426817    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="124a32e6-1529-4b84-aa39-dda6f9332364" volumeName="kubernetes.io/empty-dir/124a32e6-1529-4b84-aa39-dda6f9332364-ca-trust-extracted" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426830    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="948eb959-3a16-4ea4-a87b-9497f8bf46d9" volumeName="kubernetes.io/projected/948eb959-3a16-4ea4-a87b-9497f8bf46d9-kube-api-access-zprsb" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426863    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b93e0f22-01e7-4128-a999-da9b96e029f8" volumeName="kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-stats-auth" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426879    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb81587c-bf28-4f77-ac53-02cc794c5365" volumeName="kubernetes.io/projected/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-api-access-spj5p" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426908    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3bc3d36-b8b2-409b-8c62-66590a8e3a5c" volumeName="kubernetes.io/configmap/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-telemetry-config" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426928    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4ba5ec9-5277-460a-976e-3b1f154f7623" volumeName="kubernetes.io/secret/f4ba5ec9-5277-460a-976e-3b1f154f7623-serving-cert" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426941    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="124a32e6-1529-4b84-aa39-dda6f9332364" volumeName="kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-kube-api-access-rdn4r" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426953    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75d5ea8e-8d3d-4113-926c-5b29267d479b" volumeName="kubernetes.io/configmap/75d5ea8e-8d3d-4113-926c-5b29267d479b-cni-binary-copy" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426972    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3bc3d36-b8b2-409b-8c62-66590a8e3a5c" volumeName="kubernetes.io/projected/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-kube-api-access-qh46m" seLinuxMountContext=""
Nov 02 07:24:09.427255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.426984    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f85f18d7-a8b8-4634-af97-eb4e71dc5000" volumeName="kubernetes.io/projected/f85f18d7-a8b8-4634-af97-eb4e71dc5000-kube-api-access-rqvfj" seLinuxMountContext=""
Nov 02 07:24:09.428277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428118    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0196d04c-bd2b-48c7-bdb9-d31d83753247" volumeName="kubernetes.io/projected/0196d04c-bd2b-48c7-bdb9-d31d83753247-kube-api-access-d4pcq" seLinuxMountContext=""
Nov 02 07:24:09.428277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428147    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08bc1af6-1c3b-4740-9dc0-fa309e83b78a" volumeName="kubernetes.io/empty-dir/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-tmp-dir" seLinuxMountContext=""
Nov 02 07:24:09.428277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428160    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0" volumeName="kubernetes.io/projected/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-kube-api-access-v8jc5" seLinuxMountContext=""
Nov 02 07:24:09.428277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428172    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="531c943e-bc09-4336-9d48-065b8b716dc4" volumeName="kubernetes.io/configmap/531c943e-bc09-4336-9d48-065b8b716dc4-config" seLinuxMountContext=""
Nov 02 07:24:09.428277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428183    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af97e4d-867f-4a6c-8eb3-43d3ef4dce76" volumeName="kubernetes.io/configmap/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 07:24:09.428277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428195    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6da9cb8-1262-4288-a7fb-5991fff5c41a" volumeName="kubernetes.io/configmap/a6da9cb8-1262-4288-a7fb-5991fff5c41a-serviceca" seLinuxMountContext=""
Nov 02 07:24:09.428277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428214    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6da9cb8-1262-4288-a7fb-5991fff5c41a" volumeName="kubernetes.io/projected/a6da9cb8-1262-4288-a7fb-5991fff5c41a-kube-api-access-tfqsf" seLinuxMountContext=""
Nov 02 07:24:09.428277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428226    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b93e0f22-01e7-4128-a999-da9b96e029f8" volumeName="kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-default-certificate" seLinuxMountContext=""
Nov 02 07:24:09.428277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428238    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54a48a78-5f18-4e96-be7b-36f356cc9b39" volumeName="kubernetes.io/projected/54a48a78-5f18-4e96-be7b-36f356cc9b39-kube-api-access-k2qj8" seLinuxMountContext=""
Nov 02 07:24:09.428277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428249    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e97eaa5-7df2-484e-a200-95f3fd1114d9" volumeName="kubernetes.io/projected/7e97eaa5-7df2-484e-a200-95f3fd1114d9-kube-api-access-m2bbr" seLinuxMountContext=""
Nov 02 07:24:09.428277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428260    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bb5657a-bb11-4279-86f6-35118656d990" volumeName="kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret" seLinuxMountContext=""
Nov 02 07:24:09.428277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428273    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ff6ca66-e0d8-441f-8822-022929c12120" volumeName="kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-registry-certificates" seLinuxMountContext=""
Nov 02 07:24:09.428854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428286    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb81587c-bf28-4f77-ac53-02cc794c5365" volumeName="kubernetes.io/empty-dir/bb81587c-bf28-4f77-ac53-02cc794c5365-data-volume" seLinuxMountContext=""
Nov 02 07:24:09.428854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428297    4898 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="124a32e6-1529-4b84-aa39-dda6f9332364" volumeName="kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 07:24:09.428854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428310    4898 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 07:24:09.428854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.428318    4898 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 07:24:09.446690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.446665    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/1.log"
Nov 02 07:24:09.447811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.447781    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/1.log"
Nov 02 07:24:09.454806 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.454779    4898 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 07:24:09.456083 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.456056    4898 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 07:24:09.456083 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.456077    4898 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 07:24:09.456250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.456097    4898 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 07:24:09.456250 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:09.456154    4898 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 07:24:09.466858 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.466839    4898 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 07:24:09.467661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.467643    4898 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:24:09.467757 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.467642    4898 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:24:09.471167 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.471142    4898 generic.go:356] "Generic (PLEG): container finished" podID="0720e6d8914c83a975428ab7b77292e6" containerID="633489e80e5762c041bcd6e56687d741726ab57e6e84f48b8f4aa66edc5cddff" exitCode=0
Nov 02 07:24:09.480211 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.480183    4898 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:24:09.480802 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.480779    4898 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 07:24:09.486525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.486510    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/1.log"
Nov 02 07:24:09.486602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.486532    4898 generic.go:356] "Generic (PLEG): container finished" podID="f4ba5ec9-5277-460a-976e-3b1f154f7623" containerID="9d164f45337214cb364492d85495747f3847fda7adc1f6f41a76587105e0971b" exitCode=255
Nov 02 07:24:09.506231 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.506206    4898 generic.go:356] "Generic (PLEG): container finished" podID="7e97eaa5-7df2-484e-a200-95f3fd1114d9" containerID="64ff9ebf7a65dbf37e3f3652e2a63d713722bdbd23f9465173df0f51b07e46b3" exitCode=0
Nov 02 07:24:09.506231 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.506228    4898 generic.go:356] "Generic (PLEG): container finished" podID="7e97eaa5-7df2-484e-a200-95f3fd1114d9" containerID="d5b606a356bce51db58ab5de1375da4944a6410986ac9cb381684414a1bee06e" exitCode=0
Nov 02 07:24:09.506231 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.506235    4898 generic.go:356] "Generic (PLEG): container finished" podID="7e97eaa5-7df2-484e-a200-95f3fd1114d9" containerID="564e7b047e7de445095c5149bc08b3845876effa8acb368d3ffe86e5c291f936" exitCode=0
Nov 02 07:24:09.506408 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.506240    4898 generic.go:356] "Generic (PLEG): container finished" podID="7e97eaa5-7df2-484e-a200-95f3fd1114d9" containerID="8f8c664f78293cee4b5912f39133771d412e94507d894b5789d20ce92ba0b822" exitCode=0
Nov 02 07:24:09.506408 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.506246    4898 generic.go:356] "Generic (PLEG): container finished" podID="7e97eaa5-7df2-484e-a200-95f3fd1114d9" containerID="210ca17840dd42863fccf558cabeb47fec0e2b714bbb0c156c11b2728455d5f9" exitCode=0
Nov 02 07:24:09.506408 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.506251    4898 generic.go:356] "Generic (PLEG): container finished" podID="7e97eaa5-7df2-484e-a200-95f3fd1114d9" containerID="466e0ea0b929844ae9a6e1926438c828da35daca7977be55390bff579603c201" exitCode=0
Nov 02 07:24:09.516808 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.516790    4898 manager.go:324] Recovery completed
Nov 02 07:24:09.535480 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535460    4898 cpu_manager.go:226] "Starting" policy="none"
Nov 02 07:24:09.535480 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535474    4898 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 07:24:09.535616 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535490    4898 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 07:24:09.535616 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535593    4898 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 07:24:09.535616 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535603    4898 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 07:24:09.535700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535620    4898 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 07:24:09.535700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535625    4898 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 07:24:09.535700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535630    4898 policy_none.go:50] "Start"
Nov 02 07:24:09.535700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535638    4898 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 07:24:09.535700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535646    4898 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 07:24:09.535831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535764    4898 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 07:24:09.535831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535771    4898 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 07:24:09.535831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.535776    4898 policy_none.go:44] "Start"
Nov 02 07:24:09.541696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.541669    4898 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 07:24:09.544293 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.544275    4898 manager.go:341] "Starting Device Plugin manager"
Nov 02 07:24:09.544855 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:09.544305    4898 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 07:24:09.544953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.544857    4898 server.go:87] "Starting device plugin registration server"
Nov 02 07:24:09.545151 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.545136    4898 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 07:24:09.545229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.545152    4898 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 07:24:09.545484 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.545464    4898 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 07:24:09.545629 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.545593    4898 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 07:24:09.545711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.545637    4898 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 07:24:09.545711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.545644    4898 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 07:24:09.545821 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.545712    4898 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T07:24:09.545626755Z","UUID":"e4c0ecff-6f48-4fff-8012-e30f11a1a263","Handler":null,"Name":"","Endpoint":""}
Nov 02 07:24:09.546739 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:09.546720    4898 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 07:24:09.547166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.547147    4898 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 07:24:09.547247 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.547173    4898 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 07:24:09.556244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.556201    4898 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"]
Nov 02 07:24:09.556922 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.556906    4898 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.556962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.556933    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal" event={"ID":"0720e6d8914c83a975428ab7b77292e6","Type":"ContainerStarted","Data":"260d77242372869c493530d612d6c580dd4ef651ebf5decc2aba203d40bc3df3"}
Nov 02 07:24:09.556997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.556973    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal" event={"ID":"0720e6d8914c83a975428ab7b77292e6","Type":"ContainerDied","Data":"633489e80e5762c041bcd6e56687d741726ab57e6e84f48b8f4aa66edc5cddff"}
Nov 02 07:24:09.557034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.556996    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal" event={"ID":"0720e6d8914c83a975428ab7b77292e6","Type":"ContainerStarted","Data":"c0403c57e79b789165972e9076d6419b76ed0168408c4779079dc501d6300472"}
Nov 02 07:24:09.557070 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.557060    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal" event={"ID":"04aaad1dd656465cdf1b219bba0dba7b","Type":"ContainerStarted","Data":"57ba4bd339d969f6fcf7f3d7fab3c03699d97ec5af35807ee0fb8009dfe8f810"}
Nov 02 07:24:09.557108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.557080    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal" event={"ID":"04aaad1dd656465cdf1b219bba0dba7b","Type":"ContainerStarted","Data":"e3d627c6e8421ae277ab8773485b04ecbef13520834f978ec6dcc4191da38a1a"}
Nov 02 07:24:09.557195 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.557179    4898 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.582014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.581987    4898 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 07:24:09.582133 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:09.582036    4898 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-79.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.582587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.582575    4898 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 07:24:09.582633 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:09.582598    4898 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.646220 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.646159    4898 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 07:24:09.647401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.647383    4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 07:24:09.647464 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.647413    4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 07:24:09.647464 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.647427    4898 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-79.ec2.internal" event="NodeHasSufficientPID"
Nov 02 07:24:09.647546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.647465    4898 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.667175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.667150    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04aaad1dd656465cdf1b219bba0dba7b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-79.ec2.internal\" (UID: \"04aaad1dd656465cdf1b219bba0dba7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.667267 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.667178    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0720e6d8914c83a975428ab7b77292e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" (UID: \"0720e6d8914c83a975428ab7b77292e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.667267 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.667194    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0720e6d8914c83a975428ab7b77292e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" (UID: \"0720e6d8914c83a975428ab7b77292e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.681576 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.681554    4898 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.681653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.681616    4898 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.768100 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.768066    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04aaad1dd656465cdf1b219bba0dba7b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-79.ec2.internal\" (UID: \"04aaad1dd656465cdf1b219bba0dba7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.768100 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.768103    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0720e6d8914c83a975428ab7b77292e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" (UID: \"0720e6d8914c83a975428ab7b77292e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.768303 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.768120    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0720e6d8914c83a975428ab7b77292e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" (UID: \"0720e6d8914c83a975428ab7b77292e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.768303 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.768169    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0720e6d8914c83a975428ab7b77292e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" (UID: \"0720e6d8914c83a975428ab7b77292e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.768303 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.768179    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0720e6d8914c83a975428ab7b77292e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" (UID: \"0720e6d8914c83a975428ab7b77292e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:09.768303 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:09.768179    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04aaad1dd656465cdf1b219bba0dba7b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-79.ec2.internal\" (UID: \"04aaad1dd656465cdf1b219bba0dba7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:10.407627 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.407592    4898 apiserver.go:52] "Watching apiserver"
Nov 02 07:24:10.488051 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.488019    4898 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 07:24:10.488670 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.488647    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf","kube-system/kas-connection-checker-685895c7c9-zjjxl","openshift-multus/multus-bp5ct","openshift-network-operator/iptables-alerter-zdfn6","openshift-image-registry/image-registry-649bdd4b4b-st5hm","openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8","openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8","kube-system/konnectivity-agent-9t5tl","openshift-cluster-node-tuning-operator/tuned-rcfgw","openshift-network-console/networking-console-plugin-86c478bb88-wlj5v","kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal","kube-system/kas-connection-checker-685895c7c9-nfdng","openshift-image-registry/image-registry-769444d69f-x9wg9","openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc","openshift-console-operator/console-operator-7cd56b8dbc-tqzjh","kube-system/kas-connection-checker-685895c7c9-w6xgl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal","openshift-insights/insights-runtime-extractor-vhjhr","openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5","openshift-ingress-canary/ingress-canary-frv99","openshift-multus/multus-additional-cni-plugins-g7x6c","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc","openshift-dns/dns-default-5tv7w","kube-system/global-pull-secret-syncer-rs7kb","openshift-image-registry/node-ca-hd4ds","openshift-ovn-kubernetes/ovnkube-node-lg4p5","openshift-dns/node-resolver-k4jxw","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5","openshift-ingress/router-default-6dd94bff97-89klj","openshift-multus/network-metrics-daemon-7j98p","openshift-insights/insights-operator-7c4986d449-tct4z","openshift-network-diagnostics/network-check-target-g88mp"]
Nov 02 07:24:10.489881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.489855    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:10.490551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.490531    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:10.491142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.491111    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:10.492360 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.492341    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.493064 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.493045    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:24:10.496774 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.496743    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.497775 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.497754    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:24:10.501920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.501900    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:10.502465 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.502445    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.507241 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.507222    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 07:24:10.507397 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.507377    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 07:24:10.507397 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.507391    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 07:24:10.507502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.507398    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 07:24:10.507502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.507439    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 07:24:10.507616 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.507600    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7q455\""
Nov 02 07:24:10.507788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.507691    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.507788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.507711    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-grjcl\""
Nov 02 07:24:10.507932 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.507801    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tj7f2\""
Nov 02 07:24:10.508086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.507961    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.508086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.507999    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.508086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.508016    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.508086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.508032    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 07:24:10.508086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.508057    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.508086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.508064    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.508086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.508068    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.508347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.508095    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 07:24:10.508347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.508101    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 07:24:10.508347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.508040    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.508486 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.508469    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 07:24:10.509376 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.509363    4898 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:10.509538 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.509526    4898 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:10.522028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.522012    4898 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 07:24:10.523399 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.523374    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 07:24:10.523490 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.523380    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 07:24:10.523702 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.523683    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ztvhn\""
Nov 02 07:24:10.523759 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.523730    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-6z8kh\""
Nov 02 07:24:10.523998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.523963    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2fd7x\""
Nov 02 07:24:10.524435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.524408    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 07:24:10.524916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.524881    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 07:24:10.533104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.533083    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 07:24:10.533690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.533671    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.533773 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.533672    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.533773 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.533676    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 07:24:10.533773 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.533712    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 07:24:10.533951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.533786    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 07:24:10.533951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.533872    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.533951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.533880    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 07:24:10.533951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.533924    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.534128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.533960    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 07:24:10.534128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.534008    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 07:24:10.534128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.534120    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zjjlg\""
Nov 02 07:24:10.534765 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.534752    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jcjcp\""
Nov 02 07:24:10.535044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535026    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7x2d2\""
Nov 02 07:24:10.535145 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535043    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lsjmf\""
Nov 02 07:24:10.535145 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535114    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hdb4w\""
Nov 02 07:24:10.535145 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535131    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-szw6v\""
Nov 02 07:24:10.535255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535150    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7nffm\""
Nov 02 07:24:10.535319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535302    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c9gk6\""
Nov 02 07:24:10.535424 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535406    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-48668\""
Nov 02 07:24:10.535499 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535423    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 07:24:10.535499 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535427    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tc9dv\""
Nov 02 07:24:10.535499 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535446    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 07:24:10.535499 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535466    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.535499 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535478    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.535701 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535547    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 07:24:10.535701 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535556    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 07:24:10.535701 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535567    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfpr4\""
Nov 02 07:24:10.535701 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535585    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v4hqr\""
Nov 02 07:24:10.535701 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535630    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6xxpk\""
Nov 02 07:24:10.535917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535711    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.535917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535738    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.535917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535750    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.535917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.535740    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.536110 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536092    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 07:24:10.536234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536140    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.536234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536144    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zwv54\""
Nov 02 07:24:10.536234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536178    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 07:24:10.536234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536181    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.536234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536205    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.536234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536210    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.536234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536213    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 07:24:10.536234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536238    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536259    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536272    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536283    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536283    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zwm44\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536296    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536311    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536314    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536320    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536298    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7q84k\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536271    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536260    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lptml\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536347    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-v8lmk\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536324    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vhg4q\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536360    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536366    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jc9nv\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536353    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536391    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536407    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536442    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536449    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536470    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536520    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536548    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536557    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536580    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536572    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536587    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536599    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536605    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536615    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w2bbn\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536641    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536656    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536729    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.536756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536754    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 07:24:10.537908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536843    4898 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 07:24:10.537908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536864    4898 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 07:24:10.537908 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.536916    4898 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:10.537908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.536866    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 07:24:10.537908 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.537006    4898 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-79.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-79.ec2.internal"
Nov 02 07:24:10.537908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.537068    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 07:24:10.537908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.537149    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 07:24:10.548245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.548226    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 07:24:10.548245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.548230    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 07:24:10.571491 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.571467    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-image-registry-private-configuration\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.571596 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.571500    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-var-lib-kubelet\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.571596 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.571516    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bb81587c-bf28-4f77-ac53-02cc794c5365-crio-socket\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.571596 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.571530    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cnibin\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.571596 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.571583    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.571723 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.571690    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.571780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.571761    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-image-registry-private-configuration\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.571830 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.571815    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-node-log\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.571935 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.571918    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:24:10.571970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.571952    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-snapshots\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:24:10.572015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.571991    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-bound-sa-token\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.572015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572008    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.572129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572022    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.572129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572058    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cqp8\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-kube-api-access-9cqp8\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.572129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572084    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5bj8\" (UniqueName: \"kubernetes.io/projected/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-kube-api-access-j5bj8\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.572261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572207    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-snapshots\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:24:10.572307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572276    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sd8w\" (UniqueName: \"kubernetes.io/projected/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-kube-api-access-9sd8w\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:10.572307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572300    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.572392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572322    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-cni-bin\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.572392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572349    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhmb\" (UniqueName: \"kubernetes.io/projected/437c4525-b041-4931-ba40-99c518193657-kube-api-access-ldhmb\") pod \"kas-connection-checker-685895c7c9-w6xgl\" (UID: \"437c4525-b041-4931-ba40-99c518193657\") " pod="kube-system/kas-connection-checker-685895c7c9-w6xgl"
Nov 02 07:24:10.572392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572374    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-tmp\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:24:10.572504 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572398    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-system-cni-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.572504 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572423    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-run-netns\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.572504 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572448    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zprsb\" (UniqueName: \"kubernetes.io/projected/948eb959-3a16-4ea4-a87b-9497f8bf46d9-kube-api-access-zprsb\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.572504 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572474    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-tmp\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:24:10.572504 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572485    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-config-volume\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:10.572656 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572592    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3-agent-certs\") pod \"konnectivity-agent-9t5tl\" (UID: \"4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3\") " pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:24:10.572691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572671    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovnkube-config\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.572762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572744    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-config-volume\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:10.572801 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572784    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.572834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572804    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6da9cb8-1262-4288-a7fb-5991fff5c41a-host\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:24:10.572834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572823    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-image-registry-private-configuration\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.572910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572842    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbfdt\" (UniqueName: \"kubernetes.io/projected/531c943e-bc09-4336-9d48-065b8b716dc4-kube-api-access-vbfdt\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:24:10.572910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572878    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:24:10.572993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572920    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:10.572993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572946    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqsf\" (UniqueName: \"kubernetes.io/projected/a6da9cb8-1262-4288-a7fb-5991fff5c41a-kube-api-access-tfqsf\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:24:10.572993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572970    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-run\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.572993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.572981    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovnkube-config\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.573175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573018    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-tmp-dir\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:10.573175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573028    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3-agent-certs\") pod \"konnectivity-agent-9t5tl\" (UID: \"4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3\") " pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:24:10.573175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573037    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-image-registry-private-configuration\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.573175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573116    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-tmp-dir\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:10.573175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573123    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-k8s-cni-cncf-io\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.573175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573147    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl4ww\" (UniqueName: \"kubernetes.io/projected/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-kube-api-access-jl4ww\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:24:10.573175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573170    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fwtm\" (UniqueName: \"kubernetes.io/projected/d338565e-a9f0-44e8-ba6c-e0d88c253a10-kube-api-access-7fwtm\") pod \"volume-data-source-validator-58c9f5f46d-b2tsc\" (UID: \"d338565e-a9f0-44e8-ba6c-e0d88c253a10\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc"
Nov 02 07:24:10.573495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573186    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75d5ea8e-8d3d-4113-926c-5b29267d479b-cni-binary-copy\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.573495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573205    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-hostroot\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.573495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573219    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-conf-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.573495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573234    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54a48a78-5f18-4e96-be7b-36f356cc9b39-cert\") pod \"ingress-canary-frv99\" (UID: \"54a48a78-5f18-4e96-be7b-36f356cc9b39\") " pod="openshift-ingress-canary/ingress-canary-frv99"
Nov 02 07:24:10.573495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573283    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkxw\" (UniqueName: \"kubernetes.io/projected/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-kube-api-access-mxkxw\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:24:10.573495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573354    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-cni-netd\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.573495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573401    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-cni-multus\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.573495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573452    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54a48a78-5f18-4e96-be7b-36f356cc9b39-cert\") pod \"ingress-canary-frv99\" (UID: \"54a48a78-5f18-4e96-be7b-36f356cc9b39\") " pod="openshift-ingress-canary/ingress-canary-frv99"
Nov 02 07:24:10.573495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573456    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbdl7\" (UniqueName: \"kubernetes.io/projected/f4ba5ec9-5277-460a-976e-3b1f154f7623-kube-api-access-dbdl7\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573503    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75d5ea8e-8d3d-4113-926c-5b29267d479b-cni-binary-copy\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573567    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-modprobe-d\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573596    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg6jk\" (UniqueName: \"kubernetes.io/projected/78f8157e-9211-4f25-886e-4a9e9338e730-kube-api-access-mg6jk\") pod \"kas-connection-checker-685895c7c9-zjjxl\" (UID: \"78f8157e-9211-4f25-886e-4a9e9338e730\") " pod="kube-system/kas-connection-checker-685895c7c9-zjjxl"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573627    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-system-cni-dir\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573676    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-socket-dir-parent\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573706    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-etc-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573736    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-iptables-alerter-script\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573780    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh46m\" (UniqueName: \"kubernetes.io/projected/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-kube-api-access-qh46m\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573808    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573826    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/531c943e-bc09-4336-9d48-065b8b716dc4-config\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573843    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-service-ca-bundle\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573917    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-socket-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573936    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-sys-fs\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573963    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-serving-cert\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.573984    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:10.574012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574005    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-trusted-ca\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574019    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-service-ca-bundle\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574032    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-iptables-alerter-script\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574043    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3-konnectivity-ca\") pod \"konnectivity-agent-9t5tl\" (UID: \"4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3\") " pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574062    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574078    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qj8\" (UniqueName: \"kubernetes.io/projected/54a48a78-5f18-4e96-be7b-36f356cc9b39-kube-api-access-k2qj8\") pod \"ingress-canary-frv99\" (UID: \"54a48a78-5f18-4e96-be7b-36f356cc9b39\") " pod="openshift-ingress-canary/ingress-canary-frv99"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574083    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/531c943e-bc09-4336-9d48-065b8b716dc4-config\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574122    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxct4\" (UniqueName: \"kubernetes.io/projected/0059120b-d9ca-462c-a74b-bb99ea44c3d7-kube-api-access-gxct4\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574141    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0196d04c-bd2b-48c7-bdb9-d31d83753247-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574167    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574222    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-serving-cert\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574262    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0196d04c-bd2b-48c7-bdb9-d31d83753247-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574309    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9bb5657a-bb11-4279-86f6-35118656d990-dbus\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574354    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-slash\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574368    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3-konnectivity-ca\") pod \"konnectivity-agent-9t5tl\" (UID: \"4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3\") " pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574375    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4ba5ec9-5277-460a-976e-3b1f154f7623-trusted-ca\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574444    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-registry-certificates\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.574518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574514    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-trusted-ca\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574553    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdn4r\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-kube-api-access-rdn4r\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574566    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574582    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574607    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-registry-certificates\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574617    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb81587c-bf28-4f77-ac53-02cc794c5365-data-volume\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574643    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovn-node-metrics-cert\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574661    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0196d04c-bd2b-48c7-bdb9-d31d83753247-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574680    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-tuned\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574704    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-multus-certs\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574718    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f4ba5ec9-5277-460a-976e-3b1f154f7623-trusted-ca\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574733    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb81587c-bf28-4f77-ac53-02cc794c5365-data-volume\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574765    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-tuned\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574811    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-systemd\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574826    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovn-node-metrics-cert\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574829    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574839    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0196d04c-bd2b-48c7-bdb9-d31d83753247-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574851    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovnkube-script-lib\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574867    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574933    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-kubernetes\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574960    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/531c943e-bc09-4336-9d48-065b8b716dc4-serving-cert\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.574989    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-log-socket\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575077    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/531c943e-bc09-4336-9d48-065b8b716dc4-serving-cert\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575077    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575111    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-default-certificate\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575106    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-ovnkube-script-lib\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.575240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575136    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575263    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqpcg\" (UniqueName: \"kubernetes.io/projected/b93e0f22-01e7-4128-a999-da9b96e029f8-kube-api-access-qqpcg\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575270    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-default-certificate\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575296    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysctl-conf\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575311    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/539550af-7526-447a-b6e3-d268f758aeb1-hosts-file\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575344    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575365    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575380    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-var-lib-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575394    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575412    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jc5\" (UniqueName: \"kubernetes.io/projected/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-kube-api-access-v8jc5\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575427    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-trusted-ca\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575459    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysctl-d\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575604    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575636    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjhf\" (UniqueName: \"kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf\") pod \"network-check-target-g88mp\" (UID: \"c90ce63c-e5e9-4edf-8a1c-78ee6d092633\") " pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575662    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spj5p\" (UniqueName: \"kubernetes.io/projected/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-api-access-spj5p\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575681    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvfj\" (UniqueName: \"kubernetes.io/projected/f85f18d7-a8b8-4634-af97-eb4e71dc5000-kube-api-access-rqvfj\") pod \"kas-connection-checker-685895c7c9-nfdng\" (UID: \"f85f18d7-a8b8-4634-af97-eb4e71dc5000\") " pod="kube-system/kas-connection-checker-685895c7c9-nfdng"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575745    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-trusted-ca\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575766    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-device-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575783    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ff6ca66-e0d8-441f-8822-022929c12120-ca-trust-extracted\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575800    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-registry-certificates\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575833    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-sys\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575854    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-tmp\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575872    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm5vh\" (UniqueName: \"kubernetes.io/projected/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-kube-api-access-bm5vh\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575915    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2bbr\" (UniqueName: \"kubernetes.io/projected/7e97eaa5-7df2-484e-a200-95f3fd1114d9-kube-api-access-m2bbr\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.575948    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-registry-certificates\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576033    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-kubelet\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576044    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-tmp\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576064    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ff6ca66-e0d8-441f-8822-022929c12120-ca-trust-extracted\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576077    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-systemd\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576104    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-systemd-units\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576126    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-ovn\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576160    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576187    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576254    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-installation-pull-secrets\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.576298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576292    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-kubelet\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576431    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576468    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f4ba5ec9-5277-460a-976e-3b1f154f7623-config\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576484    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-bound-sa-token\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576571    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-stats-auth\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576591    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4ba5ec9-5277-460a-976e-3b1f154f7623-serving-cert\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576606    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-host\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576614    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f4ba5ec9-5277-460a-976e-3b1f154f7623-config\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576621    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9bb5657a-bb11-4279-86f6-35118656d990-kubelet-config\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576643    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-host-slash\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576669    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx642\" (UniqueName: \"kubernetes.io/projected/7bf60630-1d98-4928-a2bf-dda7a1e1a110-kube-api-access-rx642\") pod \"network-check-source-6f78d59f99-lrtvc\" (UID: \"7bf60630-1d98-4928-a2bf-dda7a1e1a110\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576696    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-env-overrides\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576771    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4ba5ec9-5277-460a-976e-3b1f154f7623-serving-cert\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576821    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fs4t\" (UniqueName: \"kubernetes.io/projected/7886c1fa-91a5-4ac1-9c59-98a8268bb05e-kube-api-access-9fs4t\") pod \"migrator-66df5d496b-p7hb8\" (UID: \"7886c1fa-91a5-4ac1-9c59-98a8268bb05e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576848    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-stats-auth\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576846    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576870    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/06de30c9-de01-4f63-859a-23df97972617-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576950    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/948eb959-3a16-4ea4-a87b-9497f8bf46d9-env-overrides\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.576954    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-installation-pull-secrets\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577009    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysconfig\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577084    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-installation-pull-secrets\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577133    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-cni-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577147    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/06de30c9-de01-4f63-859a-23df97972617-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577161    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4pcq\" (UniqueName: \"kubernetes.io/projected/0196d04c-bd2b-48c7-bdb9-d31d83753247-kube-api-access-d4pcq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577205    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jrmw\" (UniqueName: \"kubernetes.io/projected/75d5ea8e-8d3d-4113-926c-5b29267d479b-kube-api-access-6jrmw\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577223    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-installation-pull-secrets\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577320    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-lib-modules\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577391    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577410    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/539550af-7526-447a-b6e3-d268f758aeb1-tmp-dir\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577445    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577469    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-daemon-config\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577485    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/124a32e6-1529-4b84-aa39-dda6f9332364-ca-trust-extracted\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577540    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/539550af-7526-447a-b6e3-d268f758aeb1-tmp-dir\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577585    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/124a32e6-1529-4b84-aa39-dda6f9332364-ca-trust-extracted\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577612    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-daemon-config\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577613    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8qvr\" (UniqueName: \"kubernetes.io/projected/539550af-7526-447a-b6e3-d268f758aeb1-kube-api-access-d8qvr\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577666    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9bb5657a-bb11-4279-86f6-35118656d990-original-pull-secret\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577710    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-netns\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577733    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-cni-bin\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577736    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e97eaa5-7df2-484e-a200-95f3fd1114d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577755    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-registration-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577802    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6da9cb8-1262-4288-a7fb-5991fff5c41a-serviceca\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577842    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-cnibin\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577875    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-etc-kubernetes\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.577963    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6da9cb8-1262-4288-a7fb-5991fff5c41a-serviceca\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:24:10.597027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.597001    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbdl7\" (UniqueName: \"kubernetes.io/projected/f4ba5ec9-5277-460a-976e-3b1f154f7623-kube-api-access-dbdl7\") pod \"console-operator-7cd56b8dbc-tqzjh\" (UID: \"f4ba5ec9-5277-460a-976e-3b1f154f7623\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:10.597027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.597015    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhmb\" (UniqueName: \"kubernetes.io/projected/437c4525-b041-4931-ba40-99c518193657-kube-api-access-ldhmb\") pod \"kas-connection-checker-685895c7c9-w6xgl\" (UID: \"437c4525-b041-4931-ba40-99c518193657\") " pod="kube-system/kas-connection-checker-685895c7c9-w6xgl"
Nov 02 07:24:10.599136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.598866    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-bound-sa-token\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.599136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.598945    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2bbr\" (UniqueName: \"kubernetes.io/projected/7e97eaa5-7df2-484e-a200-95f3fd1114d9-kube-api-access-m2bbr\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.599136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.598963    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5bj8\" (UniqueName: \"kubernetes.io/projected/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-kube-api-access-j5bj8\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.599136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.598973    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxct4\" (UniqueName: \"kubernetes.io/projected/0059120b-d9ca-462c-a74b-bb99ea44c3d7-kube-api-access-gxct4\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:24:10.599136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.598949    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zprsb\" (UniqueName: \"kubernetes.io/projected/948eb959-3a16-4ea4-a87b-9497f8bf46d9-kube-api-access-zprsb\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.599136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.599079    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cqp8\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-kube-api-access-9cqp8\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.599525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.599099    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqsf\" (UniqueName: \"kubernetes.io/projected/a6da9cb8-1262-4288-a7fb-5991fff5c41a-kube-api-access-tfqsf\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:24:10.599525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.599242    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fwtm\" (UniqueName: \"kubernetes.io/projected/d338565e-a9f0-44e8-ba6c-e0d88c253a10-kube-api-access-7fwtm\") pod \"volume-data-source-validator-58c9f5f46d-b2tsc\" (UID: \"d338565e-a9f0-44e8-ba6c-e0d88c253a10\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b2tsc"
Nov 02 07:24:10.599525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.599406    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkxw\" (UniqueName: \"kubernetes.io/projected/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-kube-api-access-mxkxw\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:24:10.599525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.599412    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbfdt\" (UniqueName: \"kubernetes.io/projected/531c943e-bc09-4336-9d48-065b8b716dc4-kube-api-access-vbfdt\") pod \"service-ca-operator-544d749c8c-2khq8\" (UID: \"531c943e-bc09-4336-9d48-065b8b716dc4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8"
Nov 02 07:24:10.599727 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.599570    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jc5\" (UniqueName: \"kubernetes.io/projected/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-kube-api-access-v8jc5\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.599727 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.599625    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sd8w\" (UniqueName: \"kubernetes.io/projected/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-kube-api-access-9sd8w\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:10.599727 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.599698    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdn4r\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-kube-api-access-rdn4r\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.600385 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.600364    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx642\" (UniqueName: \"kubernetes.io/projected/7bf60630-1d98-4928-a2bf-dda7a1e1a110-kube-api-access-rx642\") pod \"network-check-source-6f78d59f99-lrtvc\" (UID: \"7bf60630-1d98-4928-a2bf-dda7a1e1a110\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lrtvc"
Nov 02 07:24:10.600569 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.600531    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm5vh\" (UniqueName: \"kubernetes.io/projected/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-kube-api-access-bm5vh\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:10.600643 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.600614    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvfj\" (UniqueName: \"kubernetes.io/projected/f85f18d7-a8b8-4634-af97-eb4e71dc5000-kube-api-access-rqvfj\") pod \"kas-connection-checker-685895c7c9-nfdng\" (UID: \"f85f18d7-a8b8-4634-af97-eb4e71dc5000\") " pod="kube-system/kas-connection-checker-685895c7c9-nfdng"
Nov 02 07:24:10.601046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.601027    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh46m\" (UniqueName: \"kubernetes.io/projected/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-kube-api-access-qh46m\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:10.601509 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.601487    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg6jk\" (UniqueName: \"kubernetes.io/projected/78f8157e-9211-4f25-886e-4a9e9338e730-kube-api-access-mg6jk\") pod \"kas-connection-checker-685895c7c9-zjjxl\" (UID: \"78f8157e-9211-4f25-886e-4a9e9338e730\") " pod="kube-system/kas-connection-checker-685895c7c9-zjjxl"
Nov 02 07:24:10.601556 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.601489    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl4ww\" (UniqueName: \"kubernetes.io/projected/9af97e4d-867f-4a6c-8eb3-43d3ef4dce76-kube-api-access-jl4ww\") pod \"insights-operator-7c4986d449-tct4z\" (UID: \"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76\") " pod="openshift-insights/insights-operator-7c4986d449-tct4z"
Nov 02 07:24:10.601644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.601630    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qj8\" (UniqueName: \"kubernetes.io/projected/54a48a78-5f18-4e96-be7b-36f356cc9b39-kube-api-access-k2qj8\") pod \"ingress-canary-frv99\" (UID: \"54a48a78-5f18-4e96-be7b-36f356cc9b39\") " pod="openshift-ingress-canary/ingress-canary-frv99"
Nov 02 07:24:10.603131 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.603115    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fs4t\" (UniqueName: \"kubernetes.io/projected/7886c1fa-91a5-4ac1-9c59-98a8268bb05e-kube-api-access-9fs4t\") pod \"migrator-66df5d496b-p7hb8\" (UID: \"7886c1fa-91a5-4ac1-9c59-98a8268bb05e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p7hb8"
Nov 02 07:24:10.603606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.603587    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqpcg\" (UniqueName: \"kubernetes.io/projected/b93e0f22-01e7-4128-a999-da9b96e029f8-kube-api-access-qqpcg\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:10.604963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.604929    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crjhf\" (UniqueName: \"kubernetes.io/projected/c90ce63c-e5e9-4edf-8a1c-78ee6d092633-kube-api-access-crjhf\") pod \"network-check-target-g88mp\" (UID: \"c90ce63c-e5e9-4edf-8a1c-78ee6d092633\") " pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:24:10.605281 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.605264    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spj5p\" (UniqueName: \"kubernetes.io/projected/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-api-access-spj5p\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.605684 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.605665    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-bound-sa-token\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.605721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.605702    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8qvr\" (UniqueName: \"kubernetes.io/projected/539550af-7526-447a-b6e3-d268f758aeb1-kube-api-access-d8qvr\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:24:10.605854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.605840    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jrmw\" (UniqueName: \"kubernetes.io/projected/75d5ea8e-8d3d-4113-926c-5b29267d479b-kube-api-access-6jrmw\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.615078 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.615056    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4pcq\" (UniqueName: \"kubernetes.io/projected/0196d04c-bd2b-48c7-bdb9-d31d83753247-kube-api-access-d4pcq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5\" (UID: \"0196d04c-bd2b-48c7-bdb9-d31d83753247\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5"
Nov 02 07:24:10.679093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679035    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-node-log\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.679093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679064    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:24:10.679293 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.679262    4898 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679083    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.679550    4898 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.679614    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls podName:bb81587c-bf28-4f77-ac53-02cc794c5365 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:11.179579616 +0000 UTC m=+1.869906623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhjhr" (UID: "bb81587c-bf28-4f77-ac53-02cc794c5365") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679636    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-node-log\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.679664    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs podName:0059120b-d9ca-462c-a74b-bb99ea44c3d7 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:11.179645245 +0000 UTC m=+1.869972201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs") pod "network-metrics-daemon-7j98p" (UID: "0059120b-d9ca-462c-a74b-bb99ea44c3d7") : secret "metrics-daemon-secret" not found
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679551    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-cni-bin\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679703    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-cni-bin\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679722    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-system-cni-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679758    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-run-netns\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679786    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-system-cni-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679820    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679836    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-run-netns\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.679863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679852    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6da9cb8-1262-4288-a7fb-5991fff5c41a-host\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679915    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6da9cb8-1262-4288-a7fb-5991fff5c41a-host\") pod \"node-ca-hd4ds\" (UID: \"a6da9cb8-1262-4288-a7fb-5991fff5c41a\") " pod="openshift-image-registry/node-ca-hd4ds"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679911    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679955    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.679989    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-run\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.680023    4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680025    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-k8s-cni-cncf-io\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.680037    4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769444d69f-x9wg9: secret "image-registry-tls" not found
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680053    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-hostroot\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680090    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-conf-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.680095    4898 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.680125    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls podName:9ff6ca66-e0d8-441f-8822-022929c12120 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:11.180103984 +0000 UTC m=+1.870430948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls") pod "image-registry-769444d69f-x9wg9" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120") : secret "image-registry-tls" not found
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680144    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-cni-netd\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680145    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-conf-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.680181    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert podName:06de30c9-de01-4f63-859a-23df97972617 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:11.180162768 +0000 UTC m=+1.870489722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wlj5v" (UID: "06de30c9-de01-4f63-859a-23df97972617") : secret "networking-console-plugin-cert" not found
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680203    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-cni-netd\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680229    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-run\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680206    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-cni-multus\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680271    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-modprobe-d\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.680292    4898 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680299    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-system-cni-dir\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680319    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-hostroot\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.680336    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls podName:e3bc3d36-b8b2-409b-8c62-66590a8e3a5c nodeName:}" failed. No retries permitted until 2026-11-02 07:24:11.180323604 +0000 UTC m=+1.870650581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bbgm5" (UID: "e3bc3d36-b8b2-409b-8c62-66590a8e3a5c") : secret "cluster-monitoring-operator-tls" not found
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680356    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-system-cni-dir\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680359    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-socket-dir-parent\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680371    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-k8s-cni-cncf-io\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680396    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-etc-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680409    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-socket-dir-parent\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680433    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-socket-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680499    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-socket-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680501    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-cni-multus\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680537    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-sys-fs\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680569    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-etc-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680573    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680587    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-sys-fs\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.680590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680539    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-modprobe-d\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.680659    4898 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.680702    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls podName:6c54280e-7b8b-4f95-bbe6-a93dd82018eb nodeName:}" failed. No retries permitted until 2026-11-02 07:24:11.180690043 +0000 UTC m=+1.871017000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-spr9r" (UID: "6c54280e-7b8b-4f95-bbe6-a93dd82018eb") : secret "samples-operator-tls" not found
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680697    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9bb5657a-bb11-4279-86f6-35118656d990-dbus\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680729    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-slash\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680760    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680836    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9bb5657a-bb11-4279-86f6-35118656d990-dbus\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680880    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-multus-certs\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680939    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-systemd\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.680974    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.680997    4898 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.681009    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.681041    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls podName:08bc1af6-1c3b-4740-9dc0-fa309e83b78a nodeName:}" failed. No retries permitted until 2026-11-02 07:24:11.181029039 +0000 UTC m=+1.871355983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls") pod "dns-default-5tv7w" (UID: "08bc1af6-1c3b-4740-9dc0-fa309e83b78a") : secret "dns-default-metrics-tls" not found
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.681115    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-multus-certs\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.681152    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-kubernetes\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.681178    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-systemd\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.681826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.681188    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-log-socket\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.683815 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.683775    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.683917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.683847    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.683917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.683885    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.683917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.683907    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-log-socket\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.683917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.683858    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-kubernetes\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.684137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.683853    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-slash\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.684137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.683947    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:10.684137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.683970    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.684137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.683981    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysctl-conf\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.684137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684034    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/539550af-7526-447a-b6e3-d268f758aeb1-hosts-file\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:24:10.684137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684070    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.684137 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.684101    4898 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 07:24:10.684137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684102    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysctl-conf\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.684137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684111    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/539550af-7526-447a-b6e3-d268f758aeb1-hosts-file\") pod \"node-resolver-k4jxw\" (UID: \"539550af-7526-447a-b6e3-d268f758aeb1\") " pod="openshift-dns/node-resolver-k4jxw"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684155    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684100    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.684252    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:11.184233418 +0000 UTC m=+1.874560360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : secret "router-metrics-certs-default" not found
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684292    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-var-lib-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684325    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684348    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysctl-d\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684375    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-var-lib-openvswitch\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684391    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684394    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684430    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-device-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.684506    4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684510    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-device-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.684518    4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649bdd4b4b-st5hm: secret "image-registry-tls" not found
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684520    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-sys\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684484    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysctl-d\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.684544 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.684551    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls podName:124a32e6-1529-4b84-aa39-dda6f9332364 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:11.184537729 +0000 UTC m=+1.874864675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls") pod "image-registry-649bdd4b4b-st5hm" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364") : secret "image-registry-tls" not found
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684564    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-sys\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.684982    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-kubelet\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685014    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-systemd\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685036    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-systemd-units\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685058    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-ovn\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685069    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-var-lib-kubelet\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685081    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685093    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-systemd-units\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685105    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-kubelet\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685096    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-systemd\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685114    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-run-ovn\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685127    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685148    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-kubelet\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.685146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685152    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-host\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685173    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:10.685178    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:11.185163872 +0000 UTC m=+1.875490833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : configmap references non-existent config key: service-ca.crt
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685203    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9bb5657a-bb11-4279-86f6-35118656d990-kubelet-config\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685210    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-host\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685223    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9bb5657a-bb11-4279-86f6-35118656d990-kubelet-config\") pod \"global-pull-secret-syncer-rs7kb\" (UID: \"9bb5657a-bb11-4279-86f6-35118656d990\") " pod="kube-system/global-pull-secret-syncer-rs7kb"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685230    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-host-slash\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685258    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysconfig\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685279    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-cni-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685297    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400-host-slash\") pod \"iptables-alerter-zdfn6\" (UID: \"05d3ac01-f3d4-4c1f-81ee-7ad6f1cc5400\") " pod="openshift-network-operator/iptables-alerter-zdfn6"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685303    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-lib-modules\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685306    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-etc-sysconfig\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685349    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-multus-cni-dir\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685351    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-netns\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685372    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-host-run-netns\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685375    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bb81587c-bf28-4f77-ac53-02cc794c5365-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685389    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-cni-bin\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685401    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-lib-modules\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685430    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-registration-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685429    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/948eb959-3a16-4ea4-a87b-9497f8bf46d9-host-cni-bin\") pod \"ovnkube-node-lg4p5\" (UID: \"948eb959-3a16-4ea4-a87b-9497f8bf46d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685454    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-cnibin\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685461    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0-registration-dir\") pod \"aws-ebs-csi-driver-node-mgjqf\" (UID: \"2a4860b8-d00a-4f6d-a8ac-7be4b73b2ea0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgjqf"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685474    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-etc-kubernetes\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685490    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-var-lib-kubelet\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685494    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-cnibin\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685506    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bb81587c-bf28-4f77-ac53-02cc794c5365-crio-socket\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685518    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d5ea8e-8d3d-4113-926c-5b29267d479b-etc-kubernetes\") pod \"multus-bp5ct\" (UID: \"75d5ea8e-8d3d-4113-926c-5b29267d479b\") " pod="openshift-multus/multus-bp5ct"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685526    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cnibin\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685548    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bb81587c-bf28-4f77-ac53-02cc794c5365-crio-socket\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685556    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a97cd1c-a0a6-4ea6-95de-ffadd05592b2-var-lib-kubelet\") pod \"tuned-rcfgw\" (UID: \"8a97cd1c-a0a6-4ea6-95de-ffadd05592b2\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfgw"
Nov 02 07:24:10.685700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.685588    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e97eaa5-7df2-484e-a200-95f3fd1114d9-cnibin\") pod \"multus-additional-cni-plugins-g7x6c\" (UID: \"7e97eaa5-7df2-484e-a200-95f3fd1114d9\") " pod="openshift-multus/multus-additional-cni-plugins-g7x6c"
Nov 02 07:24:10.793168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:10.793147    4898 scope.go:122] "RemoveContainer" containerID="9d164f45337214cb364492d85495747f3847fda7adc1f6f41a76587105e0971b"
Nov 02 07:24:11.190642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.190561    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:24:11.190642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.190600    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:11.190642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.190623    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.190640    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.190678    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.190718    4898 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.190723    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.190770    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.190743    4898 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.190838    4898 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.190849    4898 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.190782    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls podName:bb81587c-bf28-4f77-ac53-02cc794c5365 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.190769026 +0000 UTC m=+2.881095971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhjhr" (UID: "bb81587c-bf28-4f77-ac53-02cc794c5365") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.190791    4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.190920    4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769444d69f-x9wg9: secret "image-registry-tls" not found
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.190936    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls podName:08bc1af6-1c3b-4740-9dc0-fa309e83b78a nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.190919596 +0000 UTC m=+2.881246544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls") pod "dns-default-5tv7w" (UID: "08bc1af6-1c3b-4740-9dc0-fa309e83b78a") : secret "dns-default-metrics-tls" not found
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.190718    4898 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.190955    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls podName:9ff6ca66-e0d8-441f-8822-022929c12120 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.190945887 +0000 UTC m=+2.881272854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls") pod "image-registry-769444d69f-x9wg9" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120") : secret "image-registry-tls" not found
Nov 02 07:24:11.190965 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.190972    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls podName:6c54280e-7b8b-4f95-bbe6-a93dd82018eb nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.190963319 +0000 UTC m=+2.881290262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-spr9r" (UID: "6c54280e-7b8b-4f95-bbe6-a93dd82018eb") : secret "samples-operator-tls" not found
Nov 02 07:24:11.191537 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.190990    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert podName:06de30c9-de01-4f63-859a-23df97972617 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.190983979 +0000 UTC m=+2.881310922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wlj5v" (UID: "06de30c9-de01-4f63-859a-23df97972617") : secret "networking-console-plugin-cert" not found
Nov 02 07:24:11.191537 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.191006    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs podName:0059120b-d9ca-462c-a74b-bb99ea44c3d7 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.191001445 +0000 UTC m=+2.881328387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs") pod "network-metrics-daemon-7j98p" (UID: "0059120b-d9ca-462c-a74b-bb99ea44c3d7") : secret "metrics-daemon-secret" not found
Nov 02 07:24:11.191537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.191038    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:11.191537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.191070    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:11.191537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.191094    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:11.191537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.191099    4898 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 07:24:11.191537 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.191191    4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:11.191537 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.191204    4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649bdd4b4b-st5hm: secret "image-registry-tls" not found
Nov 02 07:24:11.191537 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.191208    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.191195149 +0000 UTC m=+2.881522107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : configmap references non-existent config key: service-ca.crt
Nov 02 07:24:11.191537 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.191233    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls podName:124a32e6-1529-4b84-aa39-dda6f9332364 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:12.191224155 +0000 UTC m=+2.881551116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls") pod "image-registry-649bdd4b4b-st5hm" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364") : secret "image-registry-tls" not found
Nov 02 07:24:11.193394 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.193370    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93e0f22-01e7-4128-a999-da9b96e029f8-metrics-certs\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:11.193494 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.193459    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3bc3d36-b8b2-409b-8c62-66590a8e3a5c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bbgm5\" (UID: \"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:11.260962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.260939    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k4jxw_539550af-7526-447a-b6e3-d268f758aeb1/dns-node-resolver/0.log"
Nov 02 07:24:11.390833 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.390806    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"
Nov 02 07:24:11.513473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.513446    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:24:11.513828 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.513812    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/1.log"
Nov 02 07:24:11.513879 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.513844    4898 generic.go:356] "Generic (PLEG): container finished" podID="f4ba5ec9-5277-460a-976e-3b1f154f7623" containerID="b1fd8051cb28c8207129f972aaf979175ca27d0d156df2f9352a4fd5ce85b33d" exitCode=255
Nov 02 07:24:11.513957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.513930    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh" event={"ID":"f4ba5ec9-5277-460a-976e-3b1f154f7623","Type":"ContainerDied","Data":"b1fd8051cb28c8207129f972aaf979175ca27d0d156df2f9352a4fd5ce85b33d"}
Nov 02 07:24:11.514028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.514013    4898 scope.go:122] "RemoveContainer" containerID="9d164f45337214cb364492d85495747f3847fda7adc1f6f41a76587105e0971b"
Nov 02 07:24:11.514332 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.514314    4898 scope.go:122] "RemoveContainer" containerID="b1fd8051cb28c8207129f972aaf979175ca27d0d156df2f9352a4fd5ce85b33d"
Nov 02 07:24:11.514529 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:11.514502    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tqzjh_openshift-console-operator(f4ba5ec9-5277-460a-976e-3b1f154f7623)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh" podUID="f4ba5ec9-5277-460a-976e-3b1f154f7623"
Nov 02 07:24:11.707969 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.707940    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5"]
Nov 02 07:24:11.711195 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:11.711132    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bc3d36_b8b2_409b_8c62_66590a8e3a5c.slice/crio-4d439f9bbaee5fa57003c6f8c73c3bb65cad87cf5a8138872691a24df197c4b2 WatchSource:0}: Error finding container 4d439f9bbaee5fa57003c6f8c73c3bb65cad87cf5a8138872691a24df197c4b2: Status 404 returned error can't find the container with id 4d439f9bbaee5fa57003c6f8c73c3bb65cad87cf5a8138872691a24df197c4b2
Nov 02 07:24:11.713148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.713132    4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 07:24:11.948681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:11.948645    4898 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:12.060261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.060180    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hd4ds_a6da9cb8-1262-4288-a7fb-5991fff5c41a/node-ca/0.log"
Nov 02 07:24:12.200665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.200631    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:12.200665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.200668    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:24:12.200829 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.200714    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:12.200829 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.200743    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:12.200829 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.200786    4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:12.200829 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.200800    4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769444d69f-x9wg9: secret "image-registry-tls" not found
Nov 02 07:24:12.201054 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.200852    4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:12.201054 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.200864    4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649bdd4b4b-st5hm: secret "image-registry-tls" not found
Nov 02 07:24:12.201054 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.200878    4898 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:24:12.201054 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.200856    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls podName:9ff6ca66-e0d8-441f-8822-022929c12120 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.200839351 +0000 UTC m=+4.891166310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls") pod "image-registry-769444d69f-x9wg9" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120") : secret "image-registry-tls" not found
Nov 02 07:24:12.201054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.200783    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:12.201054 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.200959    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls podName:124a32e6-1529-4b84-aa39-dda6f9332364 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.200928128 +0000 UTC m=+4.891255078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls") pod "image-registry-649bdd4b4b-st5hm" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364") : secret "image-registry-tls" not found
Nov 02 07:24:12.201054 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.200986    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls podName:08bc1af6-1c3b-4740-9dc0-fa309e83b78a nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.200974872 +0000 UTC m=+4.891301818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls") pod "dns-default-5tv7w" (UID: "08bc1af6-1c3b-4740-9dc0-fa309e83b78a") : secret "dns-default-metrics-tls" not found
Nov 02 07:24:12.201054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.201053    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:12.201417 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.201115    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:24:12.201417 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.201150    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:12.201417 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.201173    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle podName:b93e0f22-01e7-4128-a999-da9b96e029f8 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.201161136 +0000 UTC m=+4.891488077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle") pod "router-default-6dd94bff97-89klj" (UID: "b93e0f22-01e7-4128-a999-da9b96e029f8") : configmap references non-existent config key: service-ca.crt
Nov 02 07:24:12.201417 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.201224    4898 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 07:24:12.201417 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.201230    4898 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:12.201417 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.201274    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs podName:0059120b-d9ca-462c-a74b-bb99ea44c3d7 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.201260581 +0000 UTC m=+4.891587525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs") pod "network-metrics-daemon-7j98p" (UID: "0059120b-d9ca-462c-a74b-bb99ea44c3d7") : secret "metrics-daemon-secret" not found
Nov 02 07:24:12.201417 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.201288    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls podName:bb81587c-bf28-4f77-ac53-02cc794c5365 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:14.201282209 +0000 UTC m=+4.891609151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhjhr" (UID: "bb81587c-bf28-4f77-ac53-02cc794c5365") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:12.203095 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.203074    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c54280e-7b8b-4f95-bbe6-a93dd82018eb-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-spr9r\" (UID: \"6c54280e-7b8b-4f95-bbe6-a93dd82018eb\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:12.203195 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.203155    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06de30c9-de01-4f63-859a-23df97972617-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wlj5v\" (UID: \"06de30c9-de01-4f63-859a-23df97972617\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:24:12.292168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.292132    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"
Nov 02 07:24:12.294305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.294274    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"
Nov 02 07:24:12.519743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.519703    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5" event={"ID":"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c","Type":"ContainerStarted","Data":"4d439f9bbaee5fa57003c6f8c73c3bb65cad87cf5a8138872691a24df197c4b2"}
Nov 02 07:24:12.521431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.521408    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:24:12.521789 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.521771    4898 scope.go:122] "RemoveContainer" containerID="b1fd8051cb28c8207129f972aaf979175ca27d0d156df2f9352a4fd5ce85b33d"
Nov 02 07:24:12.522012 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:12.521994    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tqzjh_openshift-console-operator(f4ba5ec9-5277-460a-976e-3b1f154f7623)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh" podUID="f4ba5ec9-5277-460a-976e-3b1f154f7623"
Nov 02 07:24:12.602031 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.601954    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r"]
Nov 02 07:24:12.625668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.625633    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wlj5v"]
Nov 02 07:24:12.629309 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:12.629285    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06de30c9_de01_4f63_859a_23df97972617.slice/crio-6e79bdfdd479c2f2719cf85507b78e5ebfe902ac3b77f72ebb339f43ecff13bc WatchSource:0}: Error finding container 6e79bdfdd479c2f2719cf85507b78e5ebfe902ac3b77f72ebb339f43ecff13bc: Status 404 returned error can't find the container with id 6e79bdfdd479c2f2719cf85507b78e5ebfe902ac3b77f72ebb339f43ecff13bc
Nov 02 07:24:12.861853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:12.861760    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-frv99_54a48a78-5f18-4e96-be7b-36f356cc9b39/serve-healthcheck-canary/0.log"
Nov 02 07:24:13.261903 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:13.261872    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-p7hb8_7886c1fa-91a5-4ac1-9c59-98a8268bb05e/migrator/0.log"
Nov 02 07:24:13.464493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:13.464466    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5_0196d04c-bd2b-48c7-bdb9-d31d83753247/kube-storage-version-migrator-operator/0.log"
Nov 02 07:24:13.527255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:13.527173    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v" event={"ID":"06de30c9-de01-4f63-859a-23df97972617","Type":"ContainerStarted","Data":"6e79bdfdd479c2f2719cf85507b78e5ebfe902ac3b77f72ebb339f43ecff13bc"}
Nov 02 07:24:13.530980 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:13.530873    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r" event={"ID":"6c54280e-7b8b-4f95-bbe6-a93dd82018eb","Type":"ContainerStarted","Data":"ff51ebc644a1de8258b9ea63cdcec90142f277e41898a1d963dcfc0d01dac7dc"}
Nov 02 07:24:13.534639 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:13.534007    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bbgm5" event={"ID":"e3bc3d36-b8b2-409b-8c62-66590a8e3a5c","Type":"ContainerStarted","Data":"854bd634c67b556f97420ddf670ba87d4f7fce74cacf16fe15fcefa346a9ed80"}
Nov 02 07:24:14.220983 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:14.220946    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:24:14.221152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:14.221011    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:14.221152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:14.221044    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:14.221152 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:14.221105    4898 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 07:24:14.221318 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:14.221150    4898 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:14.221318 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:14.221162    4898 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 07:24:14.221318 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:14.221176    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs podName:0059120b-d9ca-462c-a74b-bb99ea44c3d7 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:18.221156383 +0000 UTC m=+8.911483341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs") pod "network-metrics-daemon-7j98p" (UID: "0059120b-d9ca-462c-a74b-bb99ea44c3d7") : secret "metrics-daemon-secret" not found
Nov 02 07:24:14.221318 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:14.221106    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:14.221318 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:14.221204    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls podName:08bc1af6-1c3b-4740-9dc0-fa309e83b78a nodeName:}" failed. No retries permitted until 2026-11-02 07:24:18.221188909 +0000 UTC m=+8.911515865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls") pod "dns-default-5tv7w" (UID: "08bc1af6-1c3b-4740-9dc0-fa309e83b78a") : secret "dns-default-metrics-tls" not found
Nov 02 07:24:14.221318 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:14.221216    4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:14.221318 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:14.221237    4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769444d69f-x9wg9: secret "image-registry-tls" not found
Nov 02 07:24:14.221318 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:14.221222    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls podName:bb81587c-bf28-4f77-ac53-02cc794c5365 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:18.221214776 +0000 UTC m=+8.911541720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhjhr" (UID: "bb81587c-bf28-4f77-ac53-02cc794c5365") : secret "insights-runtime-extractor-tls" not found
Nov 02 07:24:14.221318 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:14.221288    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls podName:9ff6ca66-e0d8-441f-8822-022929c12120 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:18.22127848 +0000 UTC m=+8.911605423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls") pod "image-registry-769444d69f-x9wg9" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120") : secret "image-registry-tls" not found
Nov 02 07:24:14.221713 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:14.221326    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:14.221713 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:14.221358    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:14.221713 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:14.221573    4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:14.221713 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:14.221590    4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649bdd4b4b-st5hm: secret "image-registry-tls" not found
Nov 02 07:24:14.221713 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:14.221631    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls podName:124a32e6-1529-4b84-aa39-dda6f9332364 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:18.22161744 +0000 UTC m=+8.911944410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls") pod "image-registry-649bdd4b4b-st5hm" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364") : secret "image-registry-tls" not found
Nov 02 07:24:14.221899 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:14.221871    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e0f22-01e7-4128-a999-da9b96e029f8-service-ca-bundle\") pod \"router-default-6dd94bff97-89klj\" (UID: \"b93e0f22-01e7-4128-a999-da9b96e029f8\") " pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:14.392924 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:14.390656    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:14.400370 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:14.400341    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:24:14.543866 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:14.543780    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wlj5v" event={"ID":"06de30c9-de01-4f63-859a-23df97972617","Type":"ContainerStarted","Data":"bc040be07e475c1666741f97f57e676bd15e37cfc8c70f69910949cab27228a1"}
Nov 02 07:24:14.710341 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:14.710318    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6dd94bff97-89klj"]
Nov 02 07:24:14.714037 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:14.714010    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93e0f22_01e7_4128_a999_da9b96e029f8.slice/crio-a041afb99689f4b73c93d5d010a95c7f7200746132f1e9b6ec389c82c509e9f0 WatchSource:0}: Error finding container a041afb99689f4b73c93d5d010a95c7f7200746132f1e9b6ec389c82c509e9f0: Status 404 returned error can't find the container with id a041afb99689f4b73c93d5d010a95c7f7200746132f1e9b6ec389c82c509e9f0
Nov 02 07:24:15.548168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:15.548135    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6dd94bff97-89klj" event={"ID":"b93e0f22-01e7-4128-a999-da9b96e029f8","Type":"ContainerStarted","Data":"2df80e2cf8a22b7538ceddd248a96975f87da1be8f3e0c6b2dc2f586be7c9287"}
Nov 02 07:24:15.548580 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:15.548186    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6dd94bff97-89klj" event={"ID":"b93e0f22-01e7-4128-a999-da9b96e029f8","Type":"ContainerStarted","Data":"a041afb99689f4b73c93d5d010a95c7f7200746132f1e9b6ec389c82c509e9f0"}
Nov 02 07:24:15.550036 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:15.550002    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r" event={"ID":"6c54280e-7b8b-4f95-bbe6-a93dd82018eb","Type":"ContainerStarted","Data":"9b03545c08e947dc5ffb9381a2ecf8503edce6aab393a2ba3572af597a041da7"}
Nov 02 07:24:15.550206 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:15.550050    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-spr9r" event={"ID":"6c54280e-7b8b-4f95-bbe6-a93dd82018eb","Type":"ContainerStarted","Data":"afeb72acc85e10ed004aba408e15fd6cd975ce3924c06a675eeea4f4cf495d64"}
Nov 02 07:24:16.391909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:16.391850    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:16.394520 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:16.394494    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:16.553718 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:16.553679    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:16.554718 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:16.554697    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6dd94bff97-89klj"
Nov 02 07:24:17.451021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:17.450993    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:18.258902 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.258846    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:24:18.258902 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.258905    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:18.259338 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.258929    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:18.259338 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.258961    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:18.259338 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.258989    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:18.259338 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:18.259076    4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:18.259338 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:18.259095    4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769444d69f-x9wg9: secret "image-registry-tls" not found
Nov 02 07:24:18.259338 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:18.259160    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls podName:9ff6ca66-e0d8-441f-8822-022929c12120 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:26.259142186 +0000 UTC m=+16.949469141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls") pod "image-registry-769444d69f-x9wg9" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120") : secret "image-registry-tls" not found
Nov 02 07:24:18.259338 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:18.259083    4898 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 07:24:18.259338 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:18.259174    4898 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-649bdd4b4b-st5hm: secret "image-registry-tls" not found
Nov 02 07:24:18.259338 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:18.259201    4898 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls podName:124a32e6-1529-4b84-aa39-dda6f9332364 nodeName:}" failed. No retries permitted until 2026-11-02 07:24:26.259192677 +0000 UTC m=+16.949519633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls") pod "image-registry-649bdd4b4b-st5hm" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364") : secret "image-registry-tls" not found
Nov 02 07:24:18.261352 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.261326    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059120b-d9ca-462c-a74b-bb99ea44c3d7-metrics-certs\") pod \"network-metrics-daemon-7j98p\" (UID: \"0059120b-d9ca-462c-a74b-bb99ea44c3d7\") " pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:24:18.261467 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.261326    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bb81587c-bf28-4f77-ac53-02cc794c5365-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhjhr\" (UID: \"bb81587c-bf28-4f77-ac53-02cc794c5365\") " pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:18.261467 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.261398    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08bc1af6-1c3b-4740-9dc0-fa309e83b78a-metrics-tls\") pod \"dns-default-5tv7w\" (UID: \"08bc1af6-1c3b-4740-9dc0-fa309e83b78a\") " pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:18.298502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.298475    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7j98p"
Nov 02 07:24:18.302408 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.302384    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:18.303679 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.303660    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhjhr"
Nov 02 07:24:18.601383 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.601309    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7j98p"]
Nov 02 07:24:18.606261 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:18.606236    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0059120b_d9ca_462c_a74b_bb99ea44c3d7.slice/crio-eb75ce9dce4f42fa79ead71cdf28f107eae5b1ee46d9d2f46372a37aaacc40b1 WatchSource:0}: Error finding container eb75ce9dce4f42fa79ead71cdf28f107eae5b1ee46d9d2f46372a37aaacc40b1: Status 404 returned error can't find the container with id eb75ce9dce4f42fa79ead71cdf28f107eae5b1ee46d9d2f46372a37aaacc40b1
Nov 02 07:24:18.615734 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.615711    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vhjhr"]
Nov 02 07:24:18.618311 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:18.618288    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb81587c_bf28_4f77_ac53_02cc794c5365.slice/crio-f59a05094def45bc4c2832c82ff115ef91392d8d8bfe9ee4411070c641f17897 WatchSource:0}: Error finding container f59a05094def45bc4c2832c82ff115ef91392d8d8bfe9ee4411070c641f17897: Status 404 returned error can't find the container with id f59a05094def45bc4c2832c82ff115ef91392d8d8bfe9ee4411070c641f17897
Nov 02 07:24:18.626586 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:18.626564    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5tv7w"]
Nov 02 07:24:18.629606 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:18.629588    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bc1af6_1c3b_4740_9dc0_fa309e83b78a.slice/crio-af42fdcb4ab1e77b159cf188b10cd9ccebf497d331d51472baa7d838e99617fd WatchSource:0}: Error finding container af42fdcb4ab1e77b159cf188b10cd9ccebf497d331d51472baa7d838e99617fd: Status 404 returned error can't find the container with id af42fdcb4ab1e77b159cf188b10cd9ccebf497d331d51472baa7d838e99617fd
Nov 02 07:24:19.566559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:19.566531    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7j98p" event={"ID":"0059120b-d9ca-462c-a74b-bb99ea44c3d7","Type":"ContainerStarted","Data":"e72b0c7e683590b2c8c3fcea8e58ecb89cd7cbb64fa9124843187d67ff9e1050"}
Nov 02 07:24:19.566999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:19.566584    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7j98p" event={"ID":"0059120b-d9ca-462c-a74b-bb99ea44c3d7","Type":"ContainerStarted","Data":"eb75ce9dce4f42fa79ead71cdf28f107eae5b1ee46d9d2f46372a37aaacc40b1"}
Nov 02 07:24:19.569701 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:19.569652    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhjhr" event={"ID":"bb81587c-bf28-4f77-ac53-02cc794c5365","Type":"ContainerStarted","Data":"c69c61b079f91c4f477dd33da7ad53a2618e9db9c914dff11b14c9eb9b0d0db2"}
Nov 02 07:24:19.569811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:19.569755    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhjhr" event={"ID":"bb81587c-bf28-4f77-ac53-02cc794c5365","Type":"ContainerStarted","Data":"1e029f000dade8477e0545b9c04c510fca672e3aa5df46ecde0111b95cafd332"}
Nov 02 07:24:19.569811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:19.569792    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhjhr" event={"ID":"bb81587c-bf28-4f77-ac53-02cc794c5365","Type":"ContainerStarted","Data":"f59a05094def45bc4c2832c82ff115ef91392d8d8bfe9ee4411070c641f17897"}
Nov 02 07:24:19.574381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:19.574354    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5tv7w" event={"ID":"08bc1af6-1c3b-4740-9dc0-fa309e83b78a","Type":"ContainerStarted","Data":"af42fdcb4ab1e77b159cf188b10cd9ccebf497d331d51472baa7d838e99617fd"}
Nov 02 07:24:20.581376 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:20.581340    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5tv7w" event={"ID":"08bc1af6-1c3b-4740-9dc0-fa309e83b78a","Type":"ContainerStarted","Data":"071f9fe60fa0b1b4e317850144fc6310a7ee7091514b2be069bd7960301288a1"}
Nov 02 07:24:20.584022 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:20.583979    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7j98p" event={"ID":"0059120b-d9ca-462c-a74b-bb99ea44c3d7","Type":"ContainerStarted","Data":"a08d18c1cfc245d4c8b0b13b8eda56876ee892b4d298b7d489659d062966b106"}
Nov 02 07:24:20.593946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:20.593540    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:20.593946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:20.593787    4898 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 07:24:20.788416 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:20.788381    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:20.788742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:20.788729    4898 scope.go:122] "RemoveContainer" containerID="b1fd8051cb28c8207129f972aaf979175ca27d0d156df2f9352a4fd5ce85b33d"
Nov 02 07:24:20.788935 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:20.788910    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tqzjh_openshift-console-operator(f4ba5ec9-5277-460a-976e-3b1f154f7623)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh" podUID="f4ba5ec9-5277-460a-976e-3b1f154f7623"
Nov 02 07:24:21.588955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:21.588913    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5tv7w" event={"ID":"08bc1af6-1c3b-4740-9dc0-fa309e83b78a","Type":"ContainerStarted","Data":"9f983727f8f72a5633c6606fc53bed25dd78e74ecc691c678f7a4106a945e70a"}
Nov 02 07:24:21.589410 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:21.589026    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:21.590839 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:21.590813    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhjhr" event={"ID":"bb81587c-bf28-4f77-ac53-02cc794c5365","Type":"ContainerStarted","Data":"d4a2ca680e4d00cde65b79fce886b08c9772147c35e29ae77b70232741660577"}
Nov 02 07:24:21.591280 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:21.591263    4898 scope.go:122] "RemoveContainer" containerID="b1fd8051cb28c8207129f972aaf979175ca27d0d156df2f9352a4fd5ce85b33d"
Nov 02 07:24:21.633770 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:21.633751    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9t5tl"
Nov 02 07:24:22.597244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:22.597214    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:24:22.597682 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:22.597324    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh" event={"ID":"f4ba5ec9-5277-460a-976e-3b1f154f7623","Type":"ContainerStarted","Data":"18cef116d02a18bad6b0f157148a42dcd95c2eacdd5902ff816ea82946270848"}
Nov 02 07:24:22.597954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:22.597936    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:22.602700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:22.602675    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tqzjh"
Nov 02 07:24:26.273471 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.273442    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-769444d69f-x9wg9"]
Nov 02 07:24:26.273855 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:26.273645    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-769444d69f-x9wg9" podUID="9ff6ca66-e0d8-441f-8822-022929c12120"
Nov 02 07:24:26.327414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.327381    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:26.327609 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.327456    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:26.329726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.329696    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"image-registry-649bdd4b4b-st5hm\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") " pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:26.329909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.329869    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"image-registry-769444d69f-x9wg9\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") " pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:26.397764 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.397733    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:26.613278 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.613198    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:26.620358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.620335    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:26.629582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.629558    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") pod \"9ff6ca66-e0d8-441f-8822-022929c12120\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") "
Nov 02 07:24:26.629678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.629591    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-registry-certificates\") pod \"9ff6ca66-e0d8-441f-8822-022929c12120\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") "
Nov 02 07:24:26.629678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.629624    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-kube-api-access-9cqp8\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-kube-api-access-9cqp8\") pod \"9ff6ca66-e0d8-441f-8822-022929c12120\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") "
Nov 02 07:24:26.629678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.629652    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-installation-pull-secrets\") pod \"9ff6ca66-e0d8-441f-8822-022929c12120\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") "
Nov 02 07:24:26.629803 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.629685    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-image-registry-private-configuration\") pod \"9ff6ca66-e0d8-441f-8822-022929c12120\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") "
Nov 02 07:24:26.629803 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.629722    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-trusted-ca\") pod \"9ff6ca66-e0d8-441f-8822-022929c12120\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") "
Nov 02 07:24:26.629991 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.629969    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-registry-certificates" pod "9ff6ca66-e0d8-441f-8822-022929c12120" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:24:26.630265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.630232    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-trusted-ca" pod "9ff6ca66-e0d8-441f-8822-022929c12120" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:24:26.631961 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.631934    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls" pod "9ff6ca66-e0d8-441f-8822-022929c12120" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:24:26.632084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.632060    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-kube-api-access-9cqp8" pod "9ff6ca66-e0d8-441f-8822-022929c12120" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120"). InnerVolumeSpecName "kube-api-access-9cqp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:24:26.632147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.632063    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-image-registry-private-configuration" pod "9ff6ca66-e0d8-441f-8822-022929c12120" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:24:26.632197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.632152    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-installation-pull-secrets" pod "9ff6ca66-e0d8-441f-8822-022929c12120" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:24:26.713171 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.713141    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-649bdd4b4b-st5hm"]
Nov 02 07:24:26.717408 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:26.717386    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124a32e6_1529_4b84_aa39_dda6f9332364.slice/crio-a42490af14c6f0fe96d27b1371d4b30f8647bb4d6868c734304ae3c16f253297 WatchSource:0}: Error finding container a42490af14c6f0fe96d27b1371d4b30f8647bb4d6868c734304ae3c16f253297: Status 404 returned error can't find the container with id a42490af14c6f0fe96d27b1371d4b30f8647bb4d6868c734304ae3c16f253297
Nov 02 07:24:26.730135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.730114    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-bound-sa-token\") pod \"9ff6ca66-e0d8-441f-8822-022929c12120\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") "
Nov 02 07:24:26.730204 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.730148    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff6ca66-e0d8-441f-8822-022929c12120-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ff6ca66-e0d8-441f-8822-022929c12120-ca-trust-extracted\") pod \"9ff6ca66-e0d8-441f-8822-022929c12120\" (UID: \"9ff6ca66-e0d8-441f-8822-022929c12120\") "
Nov 02 07:24:26.730266 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.730252    4898 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-registry-tls\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:24:26.730300 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.730272    4898 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-registry-certificates\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:24:26.730300 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.730287    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cqp8\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-kube-api-access-9cqp8\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:24:26.730359 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.730302    4898 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-installation-pull-secrets\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:24:26.730359 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.730318    4898 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ff6ca66-e0d8-441f-8822-022929c12120-image-registry-private-configuration\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:24:26.730359 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.730333    4898 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ff6ca66-e0d8-441f-8822-022929c12120-trusted-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:24:26.730449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.730431    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff6ca66-e0d8-441f-8822-022929c12120-ca-trust-extracted" pod "9ff6ca66-e0d8-441f-8822-022929c12120" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:24:26.732087 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.732062    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-bound-sa-token" pod "9ff6ca66-e0d8-441f-8822-022929c12120" (UID: "9ff6ca66-e0d8-441f-8822-022929c12120"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:24:26.830608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.830576    4898 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff6ca66-e0d8-441f-8822-022929c12120-bound-sa-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:24:26.830608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:26.830602    4898 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ff6ca66-e0d8-441f-8822-022929c12120-ca-trust-extracted\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:24:27.618539 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:27.618453    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm" event={"ID":"124a32e6-1529-4b84-aa39-dda6f9332364","Type":"ContainerStarted","Data":"922b9e1c461a18474c72b69e68d8d299bec63999446014f2a4f80752e940a7dc"}
Nov 02 07:24:27.619049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:27.619017    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm" event={"ID":"124a32e6-1529-4b84-aa39-dda6f9332364","Type":"ContainerStarted","Data":"a42490af14c6f0fe96d27b1371d4b30f8647bb4d6868c734304ae3c16f253297"}
Nov 02 07:24:27.619049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:27.619048    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:27.619197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:27.618463    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769444d69f-x9wg9"
Nov 02 07:24:27.738162 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:27.738135    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-769444d69f-x9wg9"]
Nov 02 07:24:27.743826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:27.743802    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-769444d69f-x9wg9"]
Nov 02 07:24:29.461852 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:29.461820    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ff6ca66-e0d8-441f-8822-022929c12120" path="/var/lib/kubelet/pods/9ff6ca66-e0d8-441f-8822-022929c12120/volumes"
Nov 02 07:24:31.599104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:31.599073    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5tv7w"
Nov 02 07:24:34.403222 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:34.403189    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g88mp"
Nov 02 07:24:35.369698 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.369662    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d96jh"]
Nov 02 07:24:35.419485 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.419450    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.434392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.434357    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 07:24:35.446484 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.446455    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 07:24:35.446592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.446490    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 07:24:35.446592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.446496    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 07:24:35.446686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.446611    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zwd5q\""
Nov 02 07:24:35.481770 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.481738    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-root\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.481949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.481778    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-metrics-client-ca\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.481949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.481797    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-wtmp\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.481949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.481879    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-sys\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.482056 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.481963    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-tls\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.482056 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.482011    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffbpt\" (UniqueName: \"kubernetes.io/projected/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-kube-api-access-ffbpt\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.482056 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.482037    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-accelerators-collector-config\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.482150 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.482058    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-textfile\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.482150 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.482108    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.582969    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-root\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583023    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-metrics-client-ca\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583052    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-wtmp\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583080    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-sys\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583080    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-root\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583113    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-tls\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583134    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffbpt\" (UniqueName: \"kubernetes.io/projected/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-kube-api-access-ffbpt\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583157    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-sys\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583162    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-accelerators-collector-config\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583208    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-textfile\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583239    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-wtmp\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583252    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583575    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-textfile\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583742    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-metrics-client-ca\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.583864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.583804    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-accelerators-collector-config\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.585591 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.585559    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-tls\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.585699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.585676    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.607321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.607290    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffbpt\" (UniqueName: \"kubernetes.io/projected/9f08529c-e64e-4af2-8ac8-80ecf5bc2906-kube-api-access-ffbpt\") pod \"node-exporter-d96jh\" (UID: \"9f08529c-e64e-4af2-8ac8-80ecf5bc2906\") " pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.769245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:35.769208    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d96jh"
Nov 02 07:24:35.802942 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:24:35.802883    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f08529c_e64e_4af2_8ac8_80ecf5bc2906.slice/crio-4610913cccc17269a9782b604baef23becc6907333b91427a9b764d091a41637 WatchSource:0}: Error finding container 4610913cccc17269a9782b604baef23becc6907333b91427a9b764d091a41637: Status 404 returned error can't find the container with id 4610913cccc17269a9782b604baef23becc6907333b91427a9b764d091a41637
Nov 02 07:24:36.653203 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:36.653162    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d96jh" event={"ID":"9f08529c-e64e-4af2-8ac8-80ecf5bc2906","Type":"ContainerStarted","Data":"4610913cccc17269a9782b604baef23becc6907333b91427a9b764d091a41637"}
Nov 02 07:24:37.659164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:37.659132    4898 generic.go:356] "Generic (PLEG): container finished" podID="9f08529c-e64e-4af2-8ac8-80ecf5bc2906" containerID="fe00c7977ba53583caa7bb8fdf93d8ab260b84aa906456cdd2fdab7a1c1ea207" exitCode=0
Nov 02 07:24:37.659611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:37.659217    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d96jh" event={"ID":"9f08529c-e64e-4af2-8ac8-80ecf5bc2906","Type":"ContainerDied","Data":"fe00c7977ba53583caa7bb8fdf93d8ab260b84aa906456cdd2fdab7a1c1ea207"}
Nov 02 07:24:38.665302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:38.665265    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d96jh" event={"ID":"9f08529c-e64e-4af2-8ac8-80ecf5bc2906","Type":"ContainerStarted","Data":"033fe44aec4199d1882f8edc60d96ae7feea1940f279525ef27f10020629847d"}
Nov 02 07:24:38.665686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:38.665318    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d96jh" event={"ID":"9f08529c-e64e-4af2-8ac8-80ecf5bc2906","Type":"ContainerStarted","Data":"36d47ef790e9f5da0e833455c72a6062954b9495a61468dbd8dbab189bd2110a"}
Nov 02 07:24:38.733647 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:38.733427    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d96jh" podStartSLOduration=2.391892012 podStartE2EDuration="3.733407896s" totalImagesPullingTime="1.26068423s" totalInitContainerRuntime="80.831654ms" isStatefulPod=true podCreationTimestamp="2026-11-02 07:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:24:38.71972626 +0000 UTC m=+29.410053224" watchObservedRunningTime="2026-11-02 07:24:38.733407896 +0000 UTC m=+29.423734862"
Nov 02 07:24:39.608991 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:39.608959    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lg4p5"
Nov 02 07:24:44.990020 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:24:44.989983    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:24:48.591881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:48.591847    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-649bdd4b4b-st5hm"]
Nov 02 07:24:48.596387 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:48.596357    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:24:59.701157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:24:59.701051    4898 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t5tl_4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3/konnectivity-agent/0.log" limit=1024
Nov 02 07:25:13.611931 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.611851    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm" podUID="124a32e6-1529-4b84-aa39-dda6f9332364" containerName="registry" containerID="cri-o://922b9e1c461a18474c72b69e68d8d299bec63999446014f2a4f80752e940a7dc" gracePeriod=30
Nov 02 07:25:13.804406 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.804365    4898 generic.go:356] "Generic (PLEG): container finished" podID="124a32e6-1529-4b84-aa39-dda6f9332364" containerID="922b9e1c461a18474c72b69e68d8d299bec63999446014f2a4f80752e940a7dc" exitCode=0
Nov 02 07:25:13.804559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.804426    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm" event={"ID":"124a32e6-1529-4b84-aa39-dda6f9332364","Type":"ContainerDied","Data":"922b9e1c461a18474c72b69e68d8d299bec63999446014f2a4f80752e940a7dc"}
Nov 02 07:25:13.854951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.854928    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:25:13.982303 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.982274    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-kube-api-access-rdn4r\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-kube-api-access-rdn4r\") pod \"124a32e6-1529-4b84-aa39-dda6f9332364\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") "
Nov 02 07:25:13.982472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.982312    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124a32e6-1529-4b84-aa39-dda6f9332364-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/124a32e6-1529-4b84-aa39-dda6f9332364-ca-trust-extracted\") pod \"124a32e6-1529-4b84-aa39-dda6f9332364\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") "
Nov 02 07:25:13.982472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.982371    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-image-registry-private-configuration\") pod \"124a32e6-1529-4b84-aa39-dda6f9332364\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") "
Nov 02 07:25:13.982472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.982390    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") pod \"124a32e6-1529-4b84-aa39-dda6f9332364\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") "
Nov 02 07:25:13.982472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.982411    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-installation-pull-secrets\") pod \"124a32e6-1529-4b84-aa39-dda6f9332364\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") "
Nov 02 07:25:13.982472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.982439    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-bound-sa-token\") pod \"124a32e6-1529-4b84-aa39-dda6f9332364\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") "
Nov 02 07:25:13.982472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.982470    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-trusted-ca\") pod \"124a32e6-1529-4b84-aa39-dda6f9332364\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") "
Nov 02 07:25:13.982768 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.982515    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-registry-certificates\") pod \"124a32e6-1529-4b84-aa39-dda6f9332364\" (UID: \"124a32e6-1529-4b84-aa39-dda6f9332364\") "
Nov 02 07:25:13.983089 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.983029    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-trusted-ca" pod "124a32e6-1529-4b84-aa39-dda6f9332364" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:25:13.983089 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.983035    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-registry-certificates" pod "124a32e6-1529-4b84-aa39-dda6f9332364" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:25:13.985368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.985185    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-installation-pull-secrets" pod "124a32e6-1529-4b84-aa39-dda6f9332364" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:25:13.985368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.985324    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-bound-sa-token" pod "124a32e6-1529-4b84-aa39-dda6f9332364" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:25:13.985368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.985338    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-image-registry-private-configuration" pod "124a32e6-1529-4b84-aa39-dda6f9332364" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:25:13.985589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.985571    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls" pod "124a32e6-1529-4b84-aa39-dda6f9332364" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:25:13.985646 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.985572    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-kube-api-access-rdn4r" pod "124a32e6-1529-4b84-aa39-dda6f9332364" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364"). InnerVolumeSpecName "kube-api-access-rdn4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:25:13.989993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:13.989970    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124a32e6-1529-4b84-aa39-dda6f9332364-ca-trust-extracted" pod "124a32e6-1529-4b84-aa39-dda6f9332364" (UID: "124a32e6-1529-4b84-aa39-dda6f9332364"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:25:14.083299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.083269    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdn4r\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-kube-api-access-rdn4r\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:25:14.083299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.083293    4898 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/124a32e6-1529-4b84-aa39-dda6f9332364-ca-trust-extracted\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:25:14.083299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.083305    4898 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-image-registry-private-configuration\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:25:14.083496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.083315    4898 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-registry-tls\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:25:14.083496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.083324    4898 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/124a32e6-1529-4b84-aa39-dda6f9332364-installation-pull-secrets\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:25:14.083496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.083332    4898 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/124a32e6-1529-4b84-aa39-dda6f9332364-bound-sa-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:25:14.083496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.083340    4898 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-trusted-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:25:14.083496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.083348    4898 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/124a32e6-1529-4b84-aa39-dda6f9332364-registry-certificates\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:25:14.809614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.809586    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm"
Nov 02 07:25:14.809614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.809611    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-649bdd4b4b-st5hm" event={"ID":"124a32e6-1529-4b84-aa39-dda6f9332364","Type":"ContainerDied","Data":"a42490af14c6f0fe96d27b1371d4b30f8647bb4d6868c734304ae3c16f253297"}
Nov 02 07:25:14.810192 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.809661    4898 scope.go:122] "RemoveContainer" containerID="922b9e1c461a18474c72b69e68d8d299bec63999446014f2a4f80752e940a7dc"
Nov 02 07:25:14.871690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.871668    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-649bdd4b4b-st5hm"]
Nov 02 07:25:14.883621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:14.883601    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-649bdd4b4b-st5hm"]
Nov 02 07:25:14.986423 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:25:14.986387    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:25:15.464617 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:15.462030    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124a32e6-1529-4b84-aa39-dda6f9332364" path="/var/lib/kubelet/pods/124a32e6-1529-4b84-aa39-dda6f9332364/volumes"
Nov 02 07:25:19.830204 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:19.830170    4898 generic.go:356] "Generic (PLEG): container finished" podID="0196d04c-bd2b-48c7-bdb9-d31d83753247" containerID="dc236a499f5c5e55d2bf536ff40f91f02f091590ff5239a280b81ee408f6cf69" exitCode=0
Nov 02 07:25:19.830732 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:19.830213    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5" event={"ID":"0196d04c-bd2b-48c7-bdb9-d31d83753247","Type":"ContainerDied","Data":"dc236a499f5c5e55d2bf536ff40f91f02f091590ff5239a280b81ee408f6cf69"}
Nov 02 07:25:19.830732 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:19.830540    4898 scope.go:122] "RemoveContainer" containerID="dc236a499f5c5e55d2bf536ff40f91f02f091590ff5239a280b81ee408f6cf69"
Nov 02 07:25:20.836092 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:20.836060    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-s4zw5" event={"ID":"0196d04c-bd2b-48c7-bdb9-d31d83753247","Type":"ContainerStarted","Data":"9c84bba3eded89fa6dae57b11ba3c1e09f44c3d49beb42f3f54e43af891ad66d"}
Nov 02 07:25:24.851799 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:24.851712    4898 generic.go:356] "Generic (PLEG): container finished" podID="9af97e4d-867f-4a6c-8eb3-43d3ef4dce76" containerID="57b2b86c17341018899b22ce895095e90fc7aedd4c5dc746a09267f74172471f" exitCode=0
Nov 02 07:25:24.852265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:24.851789    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tct4z" event={"ID":"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76","Type":"ContainerDied","Data":"57b2b86c17341018899b22ce895095e90fc7aedd4c5dc746a09267f74172471f"}
Nov 02 07:25:24.852265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:24.852162    4898 scope.go:122] "RemoveContainer" containerID="57b2b86c17341018899b22ce895095e90fc7aedd4c5dc746a09267f74172471f"
Nov 02 07:25:24.856007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:24.855985    4898 generic.go:356] "Generic (PLEG): container finished" podID="531c943e-bc09-4336-9d48-065b8b716dc4" containerID="b274e39dd2baa0fa2c4843886fcc89b04b834b10dc440ad51ed07c3cdb0edf4f" exitCode=0
Nov 02 07:25:24.856098 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:24.856026    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8" event={"ID":"531c943e-bc09-4336-9d48-065b8b716dc4","Type":"ContainerDied","Data":"b274e39dd2baa0fa2c4843886fcc89b04b834b10dc440ad51ed07c3cdb0edf4f"}
Nov 02 07:25:24.856299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:24.856284    4898 scope.go:122] "RemoveContainer" containerID="b274e39dd2baa0fa2c4843886fcc89b04b834b10dc440ad51ed07c3cdb0edf4f"
Nov 02 07:25:25.861679 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:25.861645    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2khq8" event={"ID":"531c943e-bc09-4336-9d48-065b8b716dc4","Type":"ContainerStarted","Data":"9f5566f1d94ce9d80e2420c00c42395c1a00f48e30fb40d993e7a7caa6d76af1"}
Nov 02 07:25:25.863337 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:25:25.863315    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tct4z" event={"ID":"9af97e4d-867f-4a6c-8eb3-43d3ef4dce76","Type":"ContainerStarted","Data":"66a0193a1d71f6890fa0d6c6a8aa5dcb3d1cf4704fec22e9f469e43408ddb7f4"}
Nov 02 07:25:44.988830 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:25:44.988784    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:26:14.986988 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:26:14.986950    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:26:44.986792 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:26:44.986756    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:27:14.986854 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:27:14.986816    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:27:44.987097 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:27:44.987057    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:28:14.986760 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:28:14.986677    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:28:41.099853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.099813    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"]
Nov 02 07:28:41.102319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.100157    4898 cpu_manager.go:414] "containerMap: removing container" podUID="124a32e6-1529-4b84-aa39-dda6f9332364" containerName="registry"
Nov 02 07:28:41.102319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.100173    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124a32e6-1529-4b84-aa39-dda6f9332364" containerName="registry"
Nov 02 07:28:41.102319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.100265    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="124a32e6-1529-4b84-aa39-dda6f9332364" containerName="registry"
Nov 02 07:28:41.103147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.103129    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:28:41.138152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.138126    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 07:28:41.145897 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.145866    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 07:28:41.150393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.150378    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-n6wzt\""
Nov 02 07:28:41.159043 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.159022    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"]
Nov 02 07:28:41.200831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.200800    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p\" (UID: \"6055e6de-320a-4423-8700-90cef2843272\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:28:41.200981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.200838    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p\" (UID: \"6055e6de-320a-4423-8700-90cef2843272\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:28:41.200981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.200945    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtqs6\" (UniqueName: \"kubernetes.io/projected/6055e6de-320a-4423-8700-90cef2843272-kube-api-access-vtqs6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p\" (UID: \"6055e6de-320a-4423-8700-90cef2843272\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:28:41.217821 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.217795    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"]
Nov 02 07:28:41.221102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.221087    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:28:41.242399 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.242378    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"]
Nov 02 07:28:41.246730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.246709    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-577xf\""
Nov 02 07:28:41.301590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.301562    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j\" (UID: \"66e82231-2b5c-4348-952c-7c1ec793648a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:28:41.301755 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.301607    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p\" (UID: \"6055e6de-320a-4423-8700-90cef2843272\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:28:41.301755 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.301631    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p\" (UID: \"6055e6de-320a-4423-8700-90cef2843272\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:28:41.301755 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.301713    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwd66\" (UniqueName: \"kubernetes.io/projected/66e82231-2b5c-4348-952c-7c1ec793648a-kube-api-access-rwd66\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j\" (UID: \"66e82231-2b5c-4348-952c-7c1ec793648a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:28:41.301755 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.301744    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j\" (UID: \"66e82231-2b5c-4348-952c-7c1ec793648a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:28:41.301945 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.301765    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqs6\" (UniqueName: \"kubernetes.io/projected/6055e6de-320a-4423-8700-90cef2843272-kube-api-access-vtqs6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p\" (UID: \"6055e6de-320a-4423-8700-90cef2843272\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:28:41.302074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.302055    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p\" (UID: \"6055e6de-320a-4423-8700-90cef2843272\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:28:41.302074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.302069    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p\" (UID: \"6055e6de-320a-4423-8700-90cef2843272\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:28:41.331521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.331489    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqs6\" (UniqueName: \"kubernetes.io/projected/6055e6de-320a-4423-8700-90cef2843272-kube-api-access-vtqs6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p\" (UID: \"6055e6de-320a-4423-8700-90cef2843272\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:28:41.402950 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.402915    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwd66\" (UniqueName: \"kubernetes.io/projected/66e82231-2b5c-4348-952c-7c1ec793648a-kube-api-access-rwd66\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j\" (UID: \"66e82231-2b5c-4348-952c-7c1ec793648a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:28:41.403113 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.402955    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j\" (UID: \"66e82231-2b5c-4348-952c-7c1ec793648a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:28:41.403113 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.402983    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j\" (UID: \"66e82231-2b5c-4348-952c-7c1ec793648a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:28:41.403392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.403363    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j\" (UID: \"66e82231-2b5c-4348-952c-7c1ec793648a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:28:41.403392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.403366    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j\" (UID: \"66e82231-2b5c-4348-952c-7c1ec793648a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:28:41.418108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.418088    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:28:41.426313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.426283    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwd66\" (UniqueName: \"kubernetes.io/projected/66e82231-2b5c-4348-952c-7c1ec793648a-kube-api-access-rwd66\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j\" (UID: \"66e82231-2b5c-4348-952c-7c1ec793648a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:28:41.536493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.536460    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:28:41.714156 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.714082    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"]
Nov 02 07:28:41.717781 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:28:41.717750    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6055e6de_320a_4423_8700_90cef2843272.slice/crio-b5477a96cdb238cd81303d000f5a6792cef487055753ed57b6bfa255578f72f9 WatchSource:0}: Error finding container b5477a96cdb238cd81303d000f5a6792cef487055753ed57b6bfa255578f72f9: Status 404 returned error can't find the container with id b5477a96cdb238cd81303d000f5a6792cef487055753ed57b6bfa255578f72f9
Nov 02 07:28:41.831513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:41.831333    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"]
Nov 02 07:28:41.834119 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:28:41.834092    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e82231_2b5c_4348_952c_7c1ec793648a.slice/crio-05da9ac50454bed90d99afd7e94ef532a6897eba63b6e5407026ebb791e832da WatchSource:0}: Error finding container 05da9ac50454bed90d99afd7e94ef532a6897eba63b6e5407026ebb791e832da: Status 404 returned error can't find the container with id 05da9ac50454bed90d99afd7e94ef532a6897eba63b6e5407026ebb791e832da
Nov 02 07:28:42.551753 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:42.551713    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p" event={"ID":"6055e6de-320a-4423-8700-90cef2843272","Type":"ContainerStarted","Data":"b5477a96cdb238cd81303d000f5a6792cef487055753ed57b6bfa255578f72f9"}
Nov 02 07:28:42.553182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:42.553149    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j" event={"ID":"66e82231-2b5c-4348-952c-7c1ec793648a","Type":"ContainerStarted","Data":"05da9ac50454bed90d99afd7e94ef532a6897eba63b6e5407026ebb791e832da"}
Nov 02 07:28:43.986731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:43.986689    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"]
Nov 02 07:28:43.990593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:43.990570    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:28:44.018652 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.018619    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"]
Nov 02 07:28:44.018836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.018748    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-74mnx\""
Nov 02 07:28:44.025243 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.025213    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl\" (UID: \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:28:44.025388 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.025327    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghpwn\" (UniqueName: \"kubernetes.io/projected/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-kube-api-access-ghpwn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl\" (UID: \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:28:44.025455 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.025433    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl\" (UID: \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:28:44.126372 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.125874    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl\" (UID: \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:28:44.126372 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.125962    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghpwn\" (UniqueName: \"kubernetes.io/projected/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-kube-api-access-ghpwn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl\" (UID: \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:28:44.126372 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.126022    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl\" (UID: \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:28:44.126372 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.126274    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl\" (UID: \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:28:44.126372 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.126336    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl\" (UID: \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:28:44.158503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.158464    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghpwn\" (UniqueName: \"kubernetes.io/projected/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-kube-api-access-ghpwn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl\" (UID: \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:28:44.309715 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.309627    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:28:44.604733 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:44.604646    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"]
Nov 02 07:28:44.607280 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:28:44.607244    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7c9f01_cc9c_405b_ad39_9a4f575074dd.slice/crio-bf1361e47c778fc95c24202849ae4ad456761c3445a5f55b2478c8a094275164 WatchSource:0}: Error finding container bf1361e47c778fc95c24202849ae4ad456761c3445a5f55b2478c8a094275164: Status 404 returned error can't find the container with id bf1361e47c778fc95c24202849ae4ad456761c3445a5f55b2478c8a094275164
Nov 02 07:28:44.999052 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:28:44.999014    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:28:45.572999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:45.572922    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl" event={"ID":"3c7c9f01-cc9c-405b-ad39-9a4f575074dd","Type":"ContainerStarted","Data":"bf1361e47c778fc95c24202849ae4ad456761c3445a5f55b2478c8a094275164"}
Nov 02 07:28:45.986384 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:45.986345    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"]
Nov 02 07:28:45.989987 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:45.989964    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:28:46.018215 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.018184    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"]
Nov 02 07:28:46.018707 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.018355    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-qntlz\""
Nov 02 07:28:46.042455 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.042416    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjzj6\" (UniqueName: \"kubernetes.io/projected/8436cb09-c274-40f2-add0-d527c45c9bce-kube-api-access-zjzj6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2\" (UID: \"8436cb09-c274-40f2-add0-d527c45c9bce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:28:46.042622 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.042564    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2\" (UID: \"8436cb09-c274-40f2-add0-d527c45c9bce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:28:46.042714 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.042692    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2\" (UID: \"8436cb09-c274-40f2-add0-d527c45c9bce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:28:46.084655 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.084621    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"]
Nov 02 07:28:46.088354 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.088327    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:28:46.117407 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.117381    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-x5nlj\""
Nov 02 07:28:46.127001 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.126970    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"]
Nov 02 07:28:46.146133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.144770    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2\" (UID: \"8436cb09-c274-40f2-add0-d527c45c9bce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:28:46.146133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.144831    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq\" (UID: \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:28:46.146133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.144874    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vptm\" (UniqueName: \"kubernetes.io/projected/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-kube-api-access-6vptm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq\" (UID: \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:28:46.146133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.144933    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq\" (UID: \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:28:46.146133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.144979    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2\" (UID: \"8436cb09-c274-40f2-add0-d527c45c9bce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:28:46.146133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.145021    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjzj6\" (UniqueName: \"kubernetes.io/projected/8436cb09-c274-40f2-add0-d527c45c9bce-kube-api-access-zjzj6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2\" (UID: \"8436cb09-c274-40f2-add0-d527c45c9bce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:28:46.146133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.145750    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2\" (UID: \"8436cb09-c274-40f2-add0-d527c45c9bce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:28:46.146133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.146093    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2\" (UID: \"8436cb09-c274-40f2-add0-d527c45c9bce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:28:46.171261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.171230    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjzj6\" (UniqueName: \"kubernetes.io/projected/8436cb09-c274-40f2-add0-d527c45c9bce-kube-api-access-zjzj6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2\" (UID: \"8436cb09-c274-40f2-add0-d527c45c9bce\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:28:46.180191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.180150    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"]
Nov 02 07:28:46.185249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.185225    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:28:46.203286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.203259    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-4ptvw\""
Nov 02 07:28:46.205579 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.205551    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"]
Nov 02 07:28:46.245696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.245609    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg\" (UID: \"9548a6d5-af26-4d89-8429-a6b398c120f7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:28:46.245696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.245663    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq\" (UID: \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:28:46.245966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.245699    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vptm\" (UniqueName: \"kubernetes.io/projected/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-kube-api-access-6vptm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq\" (UID: \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:28:46.245966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.245724    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbpvj\" (UniqueName: \"kubernetes.io/projected/9548a6d5-af26-4d89-8429-a6b398c120f7-kube-api-access-bbpvj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg\" (UID: \"9548a6d5-af26-4d89-8429-a6b398c120f7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:28:46.245966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.245755    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq\" (UID: \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:28:46.245966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.245802    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg\" (UID: \"9548a6d5-af26-4d89-8429-a6b398c120f7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:28:46.246288 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.246268    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq\" (UID: \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:28:46.246816 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.246793    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq\" (UID: \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:28:46.270639 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.270610    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vptm\" (UniqueName: \"kubernetes.io/projected/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-kube-api-access-6vptm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq\" (UID: \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:28:46.288372 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.288345    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"]
Nov 02 07:28:46.293015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.292377    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:28:46.307360 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.307338    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-2xsdw\""
Nov 02 07:28:46.311870 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.311849    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:28:46.313948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.313921    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"]
Nov 02 07:28:46.355293 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.355249    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n\" (UID: \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:28:46.355470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.355317    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h854\" (UniqueName: \"kubernetes.io/projected/54c45d5f-f213-4bf5-9062-e5f6d3b53358-kube-api-access-4h854\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n\" (UID: \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:28:46.355470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.355365    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg\" (UID: \"9548a6d5-af26-4d89-8429-a6b398c120f7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:28:46.355470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.355414    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbpvj\" (UniqueName: \"kubernetes.io/projected/9548a6d5-af26-4d89-8429-a6b398c120f7-kube-api-access-bbpvj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg\" (UID: \"9548a6d5-af26-4d89-8429-a6b398c120f7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:28:46.355644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.355484    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg\" (UID: \"9548a6d5-af26-4d89-8429-a6b398c120f7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:28:46.355644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.355511    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n\" (UID: \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:28:46.356164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.356141    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg\" (UID: \"9548a6d5-af26-4d89-8429-a6b398c120f7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:28:46.356719 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.356693    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg\" (UID: \"9548a6d5-af26-4d89-8429-a6b398c120f7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:28:46.384407 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.384039    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbpvj\" (UniqueName: \"kubernetes.io/projected/9548a6d5-af26-4d89-8429-a6b398c120f7-kube-api-access-bbpvj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg\" (UID: \"9548a6d5-af26-4d89-8429-a6b398c120f7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:28:46.410610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.410296    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:28:46.456754 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.456341    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n\" (UID: \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:28:46.456754 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.456399    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n\" (UID: \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:28:46.456754 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.456435    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h854\" (UniqueName: \"kubernetes.io/projected/54c45d5f-f213-4bf5-9062-e5f6d3b53358-kube-api-access-4h854\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n\" (UID: \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:28:46.457061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.456804    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n\" (UID: \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:28:46.457128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.457059    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n\" (UID: \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:28:46.481489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.481419    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h854\" (UniqueName: \"kubernetes.io/projected/54c45d5f-f213-4bf5-9062-e5f6d3b53358-kube-api-access-4h854\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n\" (UID: \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:28:46.506121 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.506046    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:28:46.611944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.611912    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:28:46.634992 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.634913    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"]
Nov 02 07:28:46.635213 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:28:46.635182    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8436cb09_c274_40f2_add0_d527c45c9bce.slice/crio-b480ee18036b3e30af65977c24b0b69130a25f09e8b46967c1b5274be77934e6 WatchSource:0}: Error finding container b480ee18036b3e30af65977c24b0b69130a25f09e8b46967c1b5274be77934e6: Status 404 returned error can't find the container with id b480ee18036b3e30af65977c24b0b69130a25f09e8b46967c1b5274be77934e6
Nov 02 07:28:46.646768 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.646605    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"]
Nov 02 07:28:46.659926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.655062    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.678776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.673661    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7cxdc\""
Nov 02 07:28:46.678776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.673760    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 07:28:46.678776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.675569    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"]
Nov 02 07:28:46.725516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.725487    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"]
Nov 02 07:28:46.728676 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:28:46.728647    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c51ac20_b1cf_4299_9a3e_130bf52f41ee.slice/crio-4f86a43065e643f9d9ddaf4c134df213f672adf83567eff6a7423d4ced01049f WatchSource:0}: Error finding container 4f86a43065e643f9d9ddaf4c134df213f672adf83567eff6a7423d4ced01049f: Status 404 returned error can't find the container with id 4f86a43065e643f9d9ddaf4c134df213f672adf83567eff6a7423d4ced01049f
Nov 02 07:28:46.759069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.758501    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.759069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.758562    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.759069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.758601    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.759069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.758640    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.759069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.758678    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.759069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.758715    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea9d7e7-103a-4e9a-8824-e62a72191cac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.759069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.758739    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea9d7e7-103a-4e9a-8824-e62a72191cac-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.759069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.758780    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea9d7e7-103a-4e9a-8824-e62a72191cac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.759069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.758816    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxgm\" (UniqueName: \"kubernetes.io/projected/dea9d7e7-103a-4e9a-8824-e62a72191cac-kube-api-access-6zxgm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.772061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.772021    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"]
Nov 02 07:28:46.775982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.775956    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.807940 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.807881    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-4bnfl\""
Nov 02 07:28:46.813935 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.813900    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"]
Nov 02 07:28:46.815143 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.815116    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"]
Nov 02 07:28:46.826000 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:28:46.825962    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9548a6d5_af26_4d89_8429_a6b398c120f7.slice/crio-39ba24d173f4ab9fa013448e6c4ef7786a0225b1611865e8df6232e346f75e54 WatchSource:0}: Error finding container 39ba24d173f4ab9fa013448e6c4ef7786a0225b1611865e8df6232e346f75e54: Status 404 returned error can't find the container with id 39ba24d173f4ab9fa013448e6c4ef7786a0225b1611865e8df6232e346f75e54
Nov 02 07:28:46.860129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860053    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.860129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860121    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.860391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860148    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.860391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860187    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.860391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860211    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.860391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860235    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.860391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860263    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6mtr\" (UniqueName: \"kubernetes.io/projected/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-kube-api-access-w6mtr\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.860391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860308    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.860391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860332    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.860391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860364    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.860780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860398    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.860780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860430    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.860780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860460    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.860780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860499    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.860780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860525    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea9d7e7-103a-4e9a-8824-e62a72191cac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.860780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860551    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea9d7e7-103a-4e9a-8824-e62a72191cac-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.860780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860600    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea9d7e7-103a-4e9a-8824-e62a72191cac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.860780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.860646    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxgm\" (UniqueName: \"kubernetes.io/projected/dea9d7e7-103a-4e9a-8824-e62a72191cac-kube-api-access-6zxgm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.861868 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.861835    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.862604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.862576    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea9d7e7-103a-4e9a-8824-e62a72191cac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.862752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.862719    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.863109 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.862946    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.863109 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.863072    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.864667 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.864643    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea9d7e7-103a-4e9a-8824-e62a72191cac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.866653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.866614    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea9d7e7-103a-4e9a-8824-e62a72191cac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.884415 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.884358    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxgm\" (UniqueName: \"kubernetes.io/projected/dea9d7e7-103a-4e9a-8824-e62a72191cac-kube-api-access-6zxgm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.885658 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.885612    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea9d7e7-103a-4e9a-8824-e62a72191cac-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pnq5t\" (UID: \"dea9d7e7-103a-4e9a-8824-e62a72191cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.936133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.935759    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"]
Nov 02 07:28:46.937552 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:28:46.937511    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c45d5f_f213_4bf5_9062_e5f6d3b53358.slice/crio-b0232aa707c8fb788014189b89a56be46590fa7bca0beab0cac33e9d591f01f6 WatchSource:0}: Error finding container b0232aa707c8fb788014189b89a56be46590fa7bca0beab0cac33e9d591f01f6: Status 404 returned error can't find the container with id b0232aa707c8fb788014189b89a56be46590fa7bca0beab0cac33e9d591f01f6
Nov 02 07:28:46.961166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.961135    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.961308 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.961183    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.962398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.961555    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.962398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.961624    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.962398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.961994    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.962398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.962020    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6mtr\" (UniqueName: \"kubernetes.io/projected/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-kube-api-access-w6mtr\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.962398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.962059    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.962398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.962082    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.962398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.962120    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.962398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.962153    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.963037 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.962755    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.963037 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.963010    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.963146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.963064    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.963288 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.963269    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.964237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.964171    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.969074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.969051    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.974853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.974831    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:46.987998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.987926    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6mtr\" (UniqueName: \"kubernetes.io/projected/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-kube-api-access-w6mtr\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:46.989130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:46.989103    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd507cc-d1ef-4a8a-92c3-8995091dd0a3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2\" (UID: \"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:47.102437 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:47.102144    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:47.295671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:47.295575    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"]
Nov 02 07:28:47.304427 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:28:47.304388    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea9d7e7_103a_4e9a_8824_e62a72191cac.slice/crio-a9c8dbf2ecc540971c606d95a61b560c1313eff9655dcd7c48d0c2028c687681 WatchSource:0}: Error finding container a9c8dbf2ecc540971c606d95a61b560c1313eff9655dcd7c48d0c2028c687681: Status 404 returned error can't find the container with id a9c8dbf2ecc540971c606d95a61b560c1313eff9655dcd7c48d0c2028c687681
Nov 02 07:28:47.432844 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:47.432748    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"]
Nov 02 07:28:47.589204 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:47.589067    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n" event={"ID":"54c45d5f-f213-4bf5-9062-e5f6d3b53358","Type":"ContainerStarted","Data":"b0232aa707c8fb788014189b89a56be46590fa7bca0beab0cac33e9d591f01f6"}
Nov 02 07:28:47.591134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:47.591047    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq" event={"ID":"2c51ac20-b1cf-4299-9a3e-130bf52f41ee","Type":"ContainerStarted","Data":"4f86a43065e643f9d9ddaf4c134df213f672adf83567eff6a7423d4ced01049f"}
Nov 02 07:28:47.594103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:47.594041    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg" event={"ID":"9548a6d5-af26-4d89-8429-a6b398c120f7","Type":"ContainerStarted","Data":"39ba24d173f4ab9fa013448e6c4ef7786a0225b1611865e8df6232e346f75e54"}
Nov 02 07:28:47.598746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:47.598629    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2" event={"ID":"8436cb09-c274-40f2-add0-d527c45c9bce","Type":"ContainerStarted","Data":"b480ee18036b3e30af65977c24b0b69130a25f09e8b46967c1b5274be77934e6"}
Nov 02 07:28:47.604840 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:47.604761    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" event={"ID":"dea9d7e7-103a-4e9a-8824-e62a72191cac","Type":"ContainerStarted","Data":"a9c8dbf2ecc540971c606d95a61b560c1313eff9655dcd7c48d0c2028c687681"}
Nov 02 07:28:47.606460 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:47.606383    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2" event={"ID":"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3","Type":"ContainerStarted","Data":"b294c67fee54e3ff843c4823a494c77a44a76c83d0845969497c2fe467d38e9f"}
Nov 02 07:28:53.672834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:53.672789    4898 generic.go:356] "Generic (PLEG): container finished" podID="66e82231-2b5c-4348-952c-7c1ec793648a" containerID="cc458313e1e927d1028e9caf1267225f8e44804da1b144fac9007b9af22012f0" exitCode=0
Nov 02 07:28:53.673325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:53.673110    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j" event={"ID":"66e82231-2b5c-4348-952c-7c1ec793648a","Type":"ContainerDied","Data":"cc458313e1e927d1028e9caf1267225f8e44804da1b144fac9007b9af22012f0"}
Nov 02 07:28:53.677260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:53.676943    4898 generic.go:356] "Generic (PLEG): container finished" podID="6055e6de-320a-4423-8700-90cef2843272" containerID="fbb203379c9029b8196be11a47ea3cff0523de6ae32d3cec09238cd77254eda0" exitCode=0
Nov 02 07:28:53.677260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:53.676995    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p" event={"ID":"6055e6de-320a-4423-8700-90cef2843272","Type":"ContainerDied","Data":"fbb203379c9029b8196be11a47ea3cff0523de6ae32d3cec09238cd77254eda0"}
Nov 02 07:28:55.280006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:55.279964    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:28:55.280431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:55.280074    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:28:55.280431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:55.280129    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:28:55.343255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:55.343214    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:28:55.343380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:55.343308    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:28:55.343380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:55.343352    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:28:55.697749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:55.696849    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" event={"ID":"dea9d7e7-103a-4e9a-8824-e62a72191cac","Type":"ContainerStarted","Data":"77a4f343749c867c13f39ed4e98cecfce7f13babf762ba275a91acb1fe3926b8"}
Nov 02 07:28:55.705643 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:55.705597    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2" event={"ID":"0bd507cc-d1ef-4a8a-92c3-8995091dd0a3","Type":"ContainerStarted","Data":"94af1b574f418a3e0b407394ed3d540c3366be0e90a11ea0beba8f3dd7f19fd4"}
Nov 02 07:28:55.801883 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:55.801822    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" podStartSLOduration=1.832654923 podStartE2EDuration="9.801801126s" totalImagesPullingTime="7.969146203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:28:55.75600021 +0000 UTC m=+286.446327186" watchObservedRunningTime="2026-11-02 07:28:55.801801126 +0000 UTC m=+286.492128090"
Nov 02 07:28:55.803310 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:55.803116    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2" podStartSLOduration=1.903031524 podStartE2EDuration="9.803107759s" totalImagesPullingTime="7.900076235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:28:55.800232823 +0000 UTC m=+286.490559788" watchObservedRunningTime="2026-11-02 07:28:55.803107759 +0000 UTC m=+286.493434724"
Nov 02 07:28:55.975752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:55.975609    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:56.103136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:56.103053    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:56.109065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:56.108729    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:56.713255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:56.713219    4898 generic.go:356] "Generic (PLEG): container finished" podID="2c51ac20-b1cf-4299-9a3e-130bf52f41ee" containerID="4f4232a063c7fd9485d9507fda85cdbf31f6f79816dd86a6c1370b770d3c4df7" exitCode=0
Nov 02 07:28:56.713705 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:56.713533    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq" event={"ID":"2c51ac20-b1cf-4299-9a3e-130bf52f41ee","Type":"ContainerDied","Data":"4f4232a063c7fd9485d9507fda85cdbf31f6f79816dd86a6c1370b770d3c4df7"}
Nov 02 07:28:56.715719 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:56.715619    4898 generic.go:356] "Generic (PLEG): container finished" podID="9548a6d5-af26-4d89-8429-a6b398c120f7" containerID="b052b7f71d906df32403a04f0422ef57257b6e76298af957f3dc03584df8d712" exitCode=0
Nov 02 07:28:56.715957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:56.715876    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg" event={"ID":"9548a6d5-af26-4d89-8429-a6b398c120f7","Type":"ContainerDied","Data":"b052b7f71d906df32403a04f0422ef57257b6e76298af957f3dc03584df8d712"}
Nov 02 07:28:56.720334 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:56.720307    4898 generic.go:356] "Generic (PLEG): container finished" podID="8436cb09-c274-40f2-add0-d527c45c9bce" containerID="de60a90524112fc628d4d9866936b9b3db83198d402593a0e9b30b63c04657b9" exitCode=0
Nov 02 07:28:56.721352 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:56.721329    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2" event={"ID":"8436cb09-c274-40f2-add0-d527c45c9bce","Type":"ContainerDied","Data":"de60a90524112fc628d4d9866936b9b3db83198d402593a0e9b30b63c04657b9"}
Nov 02 07:28:56.721452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:56.721381    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:56.722515 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:56.722442    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2"
Nov 02 07:28:56.981502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:56.981460    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:57.727583 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:57.727554    4898 generic.go:356] "Generic (PLEG): container finished" podID="54c45d5f-f213-4bf5-9062-e5f6d3b53358" containerID="6a30d6fa7c4ffc48c1b70c006cc4401eac2c1e835978d656f23dbe5955e73ef9" exitCode=0
Nov 02 07:28:57.728025 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:57.727825    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n" event={"ID":"54c45d5f-f213-4bf5-9062-e5f6d3b53358","Type":"ContainerDied","Data":"6a30d6fa7c4ffc48c1b70c006cc4401eac2c1e835978d656f23dbe5955e73ef9"}
Nov 02 07:28:57.730232 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:57.730206    4898 generic.go:356] "Generic (PLEG): container finished" podID="2c51ac20-b1cf-4299-9a3e-130bf52f41ee" containerID="4d7260f55d5d3f84d28f9fc31436eb4c9ad7ded530a48d624a2eef114f8f4800" exitCode=0
Nov 02 07:28:57.730327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:57.730288    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq" event={"ID":"2c51ac20-b1cf-4299-9a3e-130bf52f41ee","Type":"ContainerDied","Data":"4d7260f55d5d3f84d28f9fc31436eb4c9ad7ded530a48d624a2eef114f8f4800"}
Nov 02 07:28:57.731479 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:57.730831    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:57.731975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:57.731954    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:28:58.739769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:58.739735    4898 generic.go:356] "Generic (PLEG): container finished" podID="9548a6d5-af26-4d89-8429-a6b398c120f7" containerID="390ad75be3a23010b6674e4995c533967ad09fc7b2cc69b51531edf56b7f88ec" exitCode=0
Nov 02 07:28:58.740248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:58.739839    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg" event={"ID":"9548a6d5-af26-4d89-8429-a6b398c120f7","Type":"ContainerDied","Data":"390ad75be3a23010b6674e4995c533967ad09fc7b2cc69b51531edf56b7f88ec"}
Nov 02 07:28:58.744077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:58.744053    4898 generic.go:356] "Generic (PLEG): container finished" podID="8436cb09-c274-40f2-add0-d527c45c9bce" containerID="1069cc25557a4197df141ebf19fb5e70d57384d3af685ddd34f627a529aee0c0" exitCode=0
Nov 02 07:28:58.744182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:58.744140    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2" event={"ID":"8436cb09-c274-40f2-add0-d527c45c9bce","Type":"ContainerDied","Data":"1069cc25557a4197df141ebf19fb5e70d57384d3af685ddd34f627a529aee0c0"}
Nov 02 07:28:58.746694 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:58.746636    4898 generic.go:356] "Generic (PLEG): container finished" podID="66e82231-2b5c-4348-952c-7c1ec793648a" containerID="1122f908778d7d0b8a03e9b217d89f4d728f97feff73bc33da8b9d79236b8b1d" exitCode=0
Nov 02 07:28:58.746964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:58.746937    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j" event={"ID":"66e82231-2b5c-4348-952c-7c1ec793648a","Type":"ContainerDied","Data":"1122f908778d7d0b8a03e9b217d89f4d728f97feff73bc33da8b9d79236b8b1d"}
Nov 02 07:28:58.748690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:58.748671    4898 generic.go:356] "Generic (PLEG): container finished" podID="54c45d5f-f213-4bf5-9062-e5f6d3b53358" containerID="e4c7c69e4d29a0c6dcb21af33912a6128e1b5e332922ef5f009863056a795f5c" exitCode=0
Nov 02 07:28:58.748766 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:58.748746    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n" event={"ID":"54c45d5f-f213-4bf5-9062-e5f6d3b53358","Type":"ContainerDied","Data":"e4c7c69e4d29a0c6dcb21af33912a6128e1b5e332922ef5f009863056a795f5c"}
Nov 02 07:28:58.752367 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:58.752342    4898 generic.go:356] "Generic (PLEG): container finished" podID="6055e6de-320a-4423-8700-90cef2843272" containerID="ddbe77f23bc0b73a294e970cbb632eab8ed58dc4617ff4dd71db54e487dd3d79" exitCode=0
Nov 02 07:28:58.752480 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:58.752423    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p" event={"ID":"6055e6de-320a-4423-8700-90cef2843272","Type":"ContainerDied","Data":"ddbe77f23bc0b73a294e970cbb632eab8ed58dc4617ff4dd71db54e487dd3d79"}
Nov 02 07:28:58.754369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:58.754345    4898 generic.go:356] "Generic (PLEG): container finished" podID="3c7c9f01-cc9c-405b-ad39-9a4f575074dd" containerID="20efcbe69d600a8e58d85bbd773cf70468223f81988ddb1fc64524b0c0a823ed" exitCode=0
Nov 02 07:28:58.754486 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:58.754406    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl" event={"ID":"3c7c9f01-cc9c-405b-ad39-9a4f575074dd","Type":"ContainerDied","Data":"20efcbe69d600a8e58d85bbd773cf70468223f81988ddb1fc64524b0c0a823ed"}
Nov 02 07:28:59.764858 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:59.764774    4898 generic.go:356] "Generic (PLEG): container finished" podID="3c7c9f01-cc9c-405b-ad39-9a4f575074dd" containerID="1ca1b0c7dca90bfffd4d3667936e1efea1cdb49b37adf0f532090532d5e746e0" exitCode=0
Nov 02 07:28:59.766129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:28:59.766102    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl" event={"ID":"3c7c9f01-cc9c-405b-ad39-9a4f575074dd","Type":"ContainerDied","Data":"1ca1b0c7dca90bfffd4d3667936e1efea1cdb49b37adf0f532090532d5e746e0"}
Nov 02 07:29:09.577380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:09.577351    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:29:09.579385 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:09.579365    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:29:12.888923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:12.888810    4898 generic.go:356] "Generic (PLEG): container finished" podID="2c51ac20-b1cf-4299-9a3e-130bf52f41ee" containerID="c0597396f1220359a007d44d2f988a35e78d0fa0413284c8f8241609ec79a36d" exitCode=0
Nov 02 07:29:12.890225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:12.889667    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq" event={"ID":"2c51ac20-b1cf-4299-9a3e-130bf52f41ee","Type":"ContainerDied","Data":"c0597396f1220359a007d44d2f988a35e78d0fa0413284c8f8241609ec79a36d"}
Nov 02 07:29:13.898321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:13.898285    4898 generic.go:356] "Generic (PLEG): container finished" podID="54c45d5f-f213-4bf5-9062-e5f6d3b53358" containerID="0d79747c057dd6375bb7b04dd4f427d7f6297a8362f2ee697a068743bf34c624" exitCode=0
Nov 02 07:29:13.898729 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:13.898367    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n" event={"ID":"54c45d5f-f213-4bf5-9062-e5f6d3b53358","Type":"ContainerDied","Data":"0d79747c057dd6375bb7b04dd4f427d7f6297a8362f2ee697a068743bf34c624"}
Nov 02 07:29:13.900919 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:13.900868    4898 generic.go:356] "Generic (PLEG): container finished" podID="6055e6de-320a-4423-8700-90cef2843272" containerID="9336eb94cbb061f9d302fa972111672912641dd7858f15d724125584498ce872" exitCode=0
Nov 02 07:29:13.901021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:13.900913    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p" event={"ID":"6055e6de-320a-4423-8700-90cef2843272","Type":"ContainerDied","Data":"9336eb94cbb061f9d302fa972111672912641dd7858f15d724125584498ce872"}
Nov 02 07:29:13.903485 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:13.903458    4898 generic.go:356] "Generic (PLEG): container finished" podID="9548a6d5-af26-4d89-8429-a6b398c120f7" containerID="223155a765a38ba07484e25c46602dc7bc444a9b099c520cf4cfdd4fa0f2c5eb" exitCode=0
Nov 02 07:29:13.903592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:13.903540    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg" event={"ID":"9548a6d5-af26-4d89-8429-a6b398c120f7","Type":"ContainerDied","Data":"223155a765a38ba07484e25c46602dc7bc444a9b099c520cf4cfdd4fa0f2c5eb"}
Nov 02 07:29:13.905884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:13.905858    4898 generic.go:356] "Generic (PLEG): container finished" podID="8436cb09-c274-40f2-add0-d527c45c9bce" containerID="f79af5b3799015dc108df797f20a84ae14dc9c8f927d25da771ed8eb29285858" exitCode=0
Nov 02 07:29:13.905988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:13.905927    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2" event={"ID":"8436cb09-c274-40f2-add0-d527c45c9bce","Type":"ContainerDied","Data":"f79af5b3799015dc108df797f20a84ae14dc9c8f927d25da771ed8eb29285858"}
Nov 02 07:29:13.908344 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:13.908318    4898 generic.go:356] "Generic (PLEG): container finished" podID="66e82231-2b5c-4348-952c-7c1ec793648a" containerID="2bfe1ec1707329d38c9b59c4f9b4f4cba2873f2c9668c115622be2dd26bda7dc" exitCode=0
Nov 02 07:29:13.908448 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:13.908360    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j" event={"ID":"66e82231-2b5c-4348-952c-7c1ec793648a","Type":"ContainerDied","Data":"2bfe1ec1707329d38c9b59c4f9b4f4cba2873f2c9668c115622be2dd26bda7dc"}
Nov 02 07:29:14.036644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.036614    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:29:14.055293 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.055263    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-kube-api-access-6vptm\" (UniqueName: \"kubernetes.io/projected/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-kube-api-access-6vptm\") pod \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\" (UID: \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\") "
Nov 02 07:29:14.055459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.055321    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-util\" (UniqueName: \"kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-util\") pod \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\" (UID: \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\") "
Nov 02 07:29:14.055459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.055386    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-bundle\") pod \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\" (UID: \"2c51ac20-b1cf-4299-9a3e-130bf52f41ee\") "
Nov 02 07:29:14.055923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.055863    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-bundle" pod "2c51ac20-b1cf-4299-9a3e-130bf52f41ee" (UID: "2c51ac20-b1cf-4299-9a3e-130bf52f41ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:14.057592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.057565    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-kube-api-access-6vptm" pod "2c51ac20-b1cf-4299-9a3e-130bf52f41ee" (UID: "2c51ac20-b1cf-4299-9a3e-130bf52f41ee"). InnerVolumeSpecName "kube-api-access-6vptm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:29:14.061441 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.061419    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-util" pod "2c51ac20-b1cf-4299-9a3e-130bf52f41ee" (UID: "2c51ac20-b1cf-4299-9a3e-130bf52f41ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:14.156049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.156019    4898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-bundle\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:14.156049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.156051    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vptm\" (UniqueName: \"kubernetes.io/projected/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-kube-api-access-6vptm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:14.156216 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.156062    4898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c51ac20-b1cf-4299-9a3e-130bf52f41ee-util\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:14.915543 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.915501    4898 generic.go:356] "Generic (PLEG): container finished" podID="3c7c9f01-cc9c-405b-ad39-9a4f575074dd" containerID="aec432dbc3b5892017a75bf692546f1e7b547b12f28d559cb7b7fef4fd9acc2b" exitCode=0
Nov 02 07:29:14.915977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.915582    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl" event={"ID":"3c7c9f01-cc9c-405b-ad39-9a4f575074dd","Type":"ContainerDied","Data":"aec432dbc3b5892017a75bf692546f1e7b547b12f28d559cb7b7fef4fd9acc2b"}
Nov 02 07:29:14.917391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.917374    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq"
Nov 02 07:29:14.917445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.917425    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfsbq" event={"ID":"2c51ac20-b1cf-4299-9a3e-130bf52f41ee","Type":"ContainerDied","Data":"4f86a43065e643f9d9ddaf4c134df213f672adf83567eff6a7423d4ced01049f"}
Nov 02 07:29:14.917477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:14.917461    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f86a43065e643f9d9ddaf4c134df213f672adf83567eff6a7423d4ced01049f"
Nov 02 07:29:15.007252 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:29:15.007165    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:29:15.086950 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.086921    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:29:15.125848 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.125818    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:29:15.130776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.130747    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:29:15.154128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.154104    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:29:15.160414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.160395    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:29:15.163717 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.163693    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-bundle\") pod \"66e82231-2b5c-4348-952c-7c1ec793648a\" (UID: \"66e82231-2b5c-4348-952c-7c1ec793648a\") "
Nov 02 07:29:15.163778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.163746    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8436cb09-c274-40f2-add0-d527c45c9bce-kube-api-access-zjzj6\" (UniqueName: \"kubernetes.io/projected/8436cb09-c274-40f2-add0-d527c45c9bce-kube-api-access-zjzj6\") pod \"8436cb09-c274-40f2-add0-d527c45c9bce\" (UID: \"8436cb09-c274-40f2-add0-d527c45c9bce\") "
Nov 02 07:29:15.163815 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.163784    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6055e6de-320a-4423-8700-90cef2843272-kube-api-access-vtqs6\" (UniqueName: \"kubernetes.io/projected/6055e6de-320a-4423-8700-90cef2843272-kube-api-access-vtqs6\") pod \"6055e6de-320a-4423-8700-90cef2843272\" (UID: \"6055e6de-320a-4423-8700-90cef2843272\") "
Nov 02 07:29:15.163849 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.163815    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c45d5f-f213-4bf5-9062-e5f6d3b53358-kube-api-access-4h854\" (UniqueName: \"kubernetes.io/projected/54c45d5f-f213-4bf5-9062-e5f6d3b53358-kube-api-access-4h854\") pod \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\" (UID: \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\") "
Nov 02 07:29:15.163914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.163853    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e82231-2b5c-4348-952c-7c1ec793648a-kube-api-access-rwd66\" (UniqueName: \"kubernetes.io/projected/66e82231-2b5c-4348-952c-7c1ec793648a-kube-api-access-rwd66\") pod \"66e82231-2b5c-4348-952c-7c1ec793648a\" (UID: \"66e82231-2b5c-4348-952c-7c1ec793648a\") "
Nov 02 07:29:15.163914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.163883    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-util\" (UniqueName: \"kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-util\") pod \"9548a6d5-af26-4d89-8429-a6b398c120f7\" (UID: \"9548a6d5-af26-4d89-8429-a6b398c120f7\") "
Nov 02 07:29:15.167439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.163940    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-util\" (UniqueName: \"kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-util\") pod \"66e82231-2b5c-4348-952c-7c1ec793648a\" (UID: \"66e82231-2b5c-4348-952c-7c1ec793648a\") "
Nov 02 07:29:15.167439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.163966    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-util\" (UniqueName: \"kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-util\") pod \"8436cb09-c274-40f2-add0-d527c45c9bce\" (UID: \"8436cb09-c274-40f2-add0-d527c45c9bce\") "
Nov 02 07:29:15.167439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.163995    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-util\" (UniqueName: \"kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-util\") pod \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\" (UID: \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\") "
Nov 02 07:29:15.170493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.164772    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-bundle" pod "66e82231-2b5c-4348-952c-7c1ec793648a" (UID: "66e82231-2b5c-4348-952c-7c1ec793648a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:15.170628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.167154    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e82231-2b5c-4348-952c-7c1ec793648a-kube-api-access-rwd66" pod "66e82231-2b5c-4348-952c-7c1ec793648a" (UID: "66e82231-2b5c-4348-952c-7c1ec793648a"). InnerVolumeSpecName "kube-api-access-rwd66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:29:15.170628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.168117    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6055e6de-320a-4423-8700-90cef2843272-kube-api-access-vtqs6" pod "6055e6de-320a-4423-8700-90cef2843272" (UID: "6055e6de-320a-4423-8700-90cef2843272"). InnerVolumeSpecName "kube-api-access-vtqs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:29:15.170628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.170481    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-bundle\" (UniqueName: \"kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-bundle\") pod \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\" (UID: \"54c45d5f-f213-4bf5-9062-e5f6d3b53358\") "
Nov 02 07:29:15.170628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.170574    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-util\" (UniqueName: \"kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-util\") pod \"6055e6de-320a-4423-8700-90cef2843272\" (UID: \"6055e6de-320a-4423-8700-90cef2843272\") "
Nov 02 07:29:15.170790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.170630    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-bundle\") pod \"6055e6de-320a-4423-8700-90cef2843272\" (UID: \"6055e6de-320a-4423-8700-90cef2843272\") "
Nov 02 07:29:15.170790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.168134    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c45d5f-f213-4bf5-9062-e5f6d3b53358-kube-api-access-4h854" pod "54c45d5f-f213-4bf5-9062-e5f6d3b53358" (UID: "54c45d5f-f213-4bf5-9062-e5f6d3b53358"). InnerVolumeSpecName "kube-api-access-4h854". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:29:15.170790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.169564    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8436cb09-c274-40f2-add0-d527c45c9bce-kube-api-access-zjzj6" pod "8436cb09-c274-40f2-add0-d527c45c9bce" (UID: "8436cb09-c274-40f2-add0-d527c45c9bce"). InnerVolumeSpecName "kube-api-access-zjzj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:29:15.170790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.170418    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-util" pod "8436cb09-c274-40f2-add0-d527c45c9bce" (UID: "8436cb09-c274-40f2-add0-d527c45c9bce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:15.170790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.170464    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-util" pod "66e82231-2b5c-4348-952c-7c1ec793648a" (UID: "66e82231-2b5c-4348-952c-7c1ec793648a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:15.170790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.170671    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-bundle\") pod \"8436cb09-c274-40f2-add0-d527c45c9bce\" (UID: \"8436cb09-c274-40f2-add0-d527c45c9bce\") "
Nov 02 07:29:15.170790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.170755    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-bundle\") pod \"9548a6d5-af26-4d89-8429-a6b398c120f7\" (UID: \"9548a6d5-af26-4d89-8429-a6b398c120f7\") "
Nov 02 07:29:15.171021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.170804    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9548a6d5-af26-4d89-8429-a6b398c120f7-kube-api-access-bbpvj\" (UniqueName: \"kubernetes.io/projected/9548a6d5-af26-4d89-8429-a6b398c120f7-kube-api-access-bbpvj\") pod \"9548a6d5-af26-4d89-8429-a6b398c120f7\" (UID: \"9548a6d5-af26-4d89-8429-a6b398c120f7\") "
Nov 02 07:29:15.171021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.170809    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-util" pod "54c45d5f-f213-4bf5-9062-e5f6d3b53358" (UID: "54c45d5f-f213-4bf5-9062-e5f6d3b53358"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:15.171088 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171076    4898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-bundle\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.171121 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171095    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjzj6\" (UniqueName: \"kubernetes.io/projected/8436cb09-c274-40f2-add0-d527c45c9bce-kube-api-access-zjzj6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.171121 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171108    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtqs6\" (UniqueName: \"kubernetes.io/projected/6055e6de-320a-4423-8700-90cef2843272-kube-api-access-vtqs6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.171184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171123    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h854\" (UniqueName: \"kubernetes.io/projected/54c45d5f-f213-4bf5-9062-e5f6d3b53358-kube-api-access-4h854\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.171184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171136    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwd66\" (UniqueName: \"kubernetes.io/projected/66e82231-2b5c-4348-952c-7c1ec793648a-kube-api-access-rwd66\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.171184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171148    4898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66e82231-2b5c-4348-952c-7c1ec793648a-util\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.171184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171161    4898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-util\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.171184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171172    4898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-util\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.171184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171172    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-bundle" pod "8436cb09-c274-40f2-add0-d527c45c9bce" (UID: "8436cb09-c274-40f2-add0-d527c45c9bce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:15.171545 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171520    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-util" pod "9548a6d5-af26-4d89-8429-a6b398c120f7" (UID: "9548a6d5-af26-4d89-8429-a6b398c120f7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:15.171663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171641    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-bundle" pod "9548a6d5-af26-4d89-8429-a6b398c120f7" (UID: "9548a6d5-af26-4d89-8429-a6b398c120f7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:15.171709 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171667    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-bundle" pod "6055e6de-320a-4423-8700-90cef2843272" (UID: "6055e6de-320a-4423-8700-90cef2843272"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:15.171709 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.171680    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-bundle" pod "54c45d5f-f213-4bf5-9062-e5f6d3b53358" (UID: "54c45d5f-f213-4bf5-9062-e5f6d3b53358"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:15.172931 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.172912    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9548a6d5-af26-4d89-8429-a6b398c120f7-kube-api-access-bbpvj" pod "9548a6d5-af26-4d89-8429-a6b398c120f7" (UID: "9548a6d5-af26-4d89-8429-a6b398c120f7"). InnerVolumeSpecName "kube-api-access-bbpvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:29:15.176147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.176125    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-util" pod "6055e6de-320a-4423-8700-90cef2843272" (UID: "6055e6de-320a-4423-8700-90cef2843272"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:15.271992 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.271954    4898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-bundle\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.271992 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.271986    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbpvj\" (UniqueName: \"kubernetes.io/projected/9548a6d5-af26-4d89-8429-a6b398c120f7-kube-api-access-bbpvj\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.272216 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.272000    4898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9548a6d5-af26-4d89-8429-a6b398c120f7-util\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.272216 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.272013    4898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54c45d5f-f213-4bf5-9062-e5f6d3b53358-bundle\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.272216 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.272023    4898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-util\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.272216 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.272033    4898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6055e6de-320a-4423-8700-90cef2843272-bundle\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.272216 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.272043    4898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8436cb09-c274-40f2-add0-d527c45c9bce-bundle\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:15.923903 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.923852    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p"
Nov 02 07:29:15.924302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.923882    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bnw9p" event={"ID":"6055e6de-320a-4423-8700-90cef2843272","Type":"ContainerDied","Data":"b5477a96cdb238cd81303d000f5a6792cef487055753ed57b6bfa255578f72f9"}
Nov 02 07:29:15.924302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.923957    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5477a96cdb238cd81303d000f5a6792cef487055753ed57b6bfa255578f72f9"
Nov 02 07:29:15.925960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.925923    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg" event={"ID":"9548a6d5-af26-4d89-8429-a6b398c120f7","Type":"ContainerDied","Data":"39ba24d173f4ab9fa013448e6c4ef7786a0225b1611865e8df6232e346f75e54"}
Nov 02 07:29:15.925960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.925939    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sqqzg"
Nov 02 07:29:15.926156 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.925964    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39ba24d173f4ab9fa013448e6c4ef7786a0225b1611865e8df6232e346f75e54"
Nov 02 07:29:15.928118 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.928095    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2" event={"ID":"8436cb09-c274-40f2-add0-d527c45c9bce","Type":"ContainerDied","Data":"b480ee18036b3e30af65977c24b0b69130a25f09e8b46967c1b5274be77934e6"}
Nov 02 07:29:15.928213 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.928135    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b480ee18036b3e30af65977c24b0b69130a25f09e8b46967c1b5274be77934e6"
Nov 02 07:29:15.928213 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.928137    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tdtj2"
Nov 02 07:29:15.930262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.930243    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j"
Nov 02 07:29:15.930369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.930240    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cppp8j" event={"ID":"66e82231-2b5c-4348-952c-7c1ec793648a","Type":"ContainerDied","Data":"05da9ac50454bed90d99afd7e94ef532a6897eba63b6e5407026ebb791e832da"}
Nov 02 07:29:15.930369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.930358    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05da9ac50454bed90d99afd7e94ef532a6897eba63b6e5407026ebb791e832da"
Nov 02 07:29:15.932513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.932495    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n"
Nov 02 07:29:15.932809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.932795    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqqh2n" event={"ID":"54c45d5f-f213-4bf5-9062-e5f6d3b53358","Type":"ContainerDied","Data":"b0232aa707c8fb788014189b89a56be46590fa7bca0beab0cac33e9d591f01f6"}
Nov 02 07:29:15.932867 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:15.932818    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0232aa707c8fb788014189b89a56be46590fa7bca0beab0cac33e9d591f01f6"
Nov 02 07:29:16.061191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.061168    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:29:16.078097 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.078074    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-bundle\") pod \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\" (UID: \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\") "
Nov 02 07:29:16.078228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.078144    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-kube-api-access-ghpwn\" (UniqueName: \"kubernetes.io/projected/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-kube-api-access-ghpwn\") pod \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\" (UID: \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\") "
Nov 02 07:29:16.078228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.078170    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-util\" (UniqueName: \"kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-util\") pod \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\" (UID: \"3c7c9f01-cc9c-405b-ad39-9a4f575074dd\") "
Nov 02 07:29:16.078571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.078548    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-bundle" pod "3c7c9f01-cc9c-405b-ad39-9a4f575074dd" (UID: "3c7c9f01-cc9c-405b-ad39-9a4f575074dd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:16.080261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.080232    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-kube-api-access-ghpwn" pod "3c7c9f01-cc9c-405b-ad39-9a4f575074dd" (UID: "3c7c9f01-cc9c-405b-ad39-9a4f575074dd"). InnerVolumeSpecName "kube-api-access-ghpwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:29:16.083514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.083492    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-util" pod "3c7c9f01-cc9c-405b-ad39-9a4f575074dd" (UID: "3c7c9f01-cc9c-405b-ad39-9a4f575074dd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:29:16.179008 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.178927    4898 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-bundle\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:16.179008 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.178955    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghpwn\" (UniqueName: \"kubernetes.io/projected/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-kube-api-access-ghpwn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:16.179008 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.178966    4898 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c7c9f01-cc9c-405b-ad39-9a4f575074dd-util\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:29:16.883532 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883497    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq"]
Nov 02 07:29:16.883818 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883805    4898 cpu_manager.go:414] "containerMap: removing container" podUID="66e82231-2b5c-4348-952c-7c1ec793648a" containerName="pull"
Nov 02 07:29:16.883880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883820    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e82231-2b5c-4348-952c-7c1ec793648a" containerName="pull"
Nov 02 07:29:16.883880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883832    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3c7c9f01-cc9c-405b-ad39-9a4f575074dd" containerName="pull"
Nov 02 07:29:16.883880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883837    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7c9f01-cc9c-405b-ad39-9a4f575074dd" containerName="pull"
Nov 02 07:29:16.883880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883842    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3c7c9f01-cc9c-405b-ad39-9a4f575074dd" containerName="extract"
Nov 02 07:29:16.883880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883847    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7c9f01-cc9c-405b-ad39-9a4f575074dd" containerName="extract"
Nov 02 07:29:16.883880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883855    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2c51ac20-b1cf-4299-9a3e-130bf52f41ee" containerName="extract"
Nov 02 07:29:16.883880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883860    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c51ac20-b1cf-4299-9a3e-130bf52f41ee" containerName="extract"
Nov 02 07:29:16.883880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883868    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6055e6de-320a-4423-8700-90cef2843272" containerName="pull"
Nov 02 07:29:16.883880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883873    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6055e6de-320a-4423-8700-90cef2843272" containerName="pull"
Nov 02 07:29:16.883880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883880    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3c7c9f01-cc9c-405b-ad39-9a4f575074dd" containerName="util"
Nov 02 07:29:16.883880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883885    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7c9f01-cc9c-405b-ad39-9a4f575074dd" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883903    4898 cpu_manager.go:414] "containerMap: removing container" podUID="54c45d5f-f213-4bf5-9062-e5f6d3b53358" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883908    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c45d5f-f213-4bf5-9062-e5f6d3b53358" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883921    4898 cpu_manager.go:414] "containerMap: removing container" podUID="8436cb09-c274-40f2-add0-d527c45c9bce" containerName="pull"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883926    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8436cb09-c274-40f2-add0-d527c45c9bce" containerName="pull"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883933    4898 cpu_manager.go:414] "containerMap: removing container" podUID="66e82231-2b5c-4348-952c-7c1ec793648a" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883938    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e82231-2b5c-4348-952c-7c1ec793648a" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883947    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2c51ac20-b1cf-4299-9a3e-130bf52f41ee" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883952    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c51ac20-b1cf-4299-9a3e-130bf52f41ee" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883959    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2c51ac20-b1cf-4299-9a3e-130bf52f41ee" containerName="pull"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883963    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c51ac20-b1cf-4299-9a3e-130bf52f41ee" containerName="pull"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883969    4898 cpu_manager.go:414] "containerMap: removing container" podUID="54c45d5f-f213-4bf5-9062-e5f6d3b53358" containerName="pull"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883973    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c45d5f-f213-4bf5-9062-e5f6d3b53358" containerName="pull"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883985    4898 cpu_manager.go:414] "containerMap: removing container" podUID="8436cb09-c274-40f2-add0-d527c45c9bce" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883988    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8436cb09-c274-40f2-add0-d527c45c9bce" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.883996    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6055e6de-320a-4423-8700-90cef2843272" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884000    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6055e6de-320a-4423-8700-90cef2843272" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884005    4898 cpu_manager.go:414] "containerMap: removing container" podUID="66e82231-2b5c-4348-952c-7c1ec793648a" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884009    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e82231-2b5c-4348-952c-7c1ec793648a" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884015    4898 cpu_manager.go:414] "containerMap: removing container" podUID="8436cb09-c274-40f2-add0-d527c45c9bce" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884020    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8436cb09-c274-40f2-add0-d527c45c9bce" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884027    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9548a6d5-af26-4d89-8429-a6b398c120f7" containerName="pull"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884031    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9548a6d5-af26-4d89-8429-a6b398c120f7" containerName="pull"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884037    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9548a6d5-af26-4d89-8429-a6b398c120f7" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884041    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9548a6d5-af26-4d89-8429-a6b398c120f7" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884058    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6055e6de-320a-4423-8700-90cef2843272" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884062    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6055e6de-320a-4423-8700-90cef2843272" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884068    4898 cpu_manager.go:414] "containerMap: removing container" podUID="54c45d5f-f213-4bf5-9062-e5f6d3b53358" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884072    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c45d5f-f213-4bf5-9062-e5f6d3b53358" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884079    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9548a6d5-af26-4d89-8429-a6b398c120f7" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884083    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9548a6d5-af26-4d89-8429-a6b398c120f7" containerName="util"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884134    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7c9f01-cc9c-405b-ad39-9a4f575074dd" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884141    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c51ac20-b1cf-4299-9a3e-130bf52f41ee" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884147    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c45d5f-f213-4bf5-9062-e5f6d3b53358" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884154    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e82231-2b5c-4348-952c-7c1ec793648a" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884163    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8436cb09-c274-40f2-add0-d527c45c9bce" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884169    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9548a6d5-af26-4d89-8429-a6b398c120f7" containerName="extract"
Nov 02 07:29:16.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.884173    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6055e6de-320a-4423-8700-90cef2843272" containerName="extract"
Nov 02 07:29:16.887352 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.887327    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq"
Nov 02 07:29:16.903239 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.903217    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:29:16.903239 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.903217    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-b2fpm\""
Nov 02 07:29:16.903394 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.903223    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 02 07:29:16.917233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.917209    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq"]
Nov 02 07:29:16.939602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.939573    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl"
Nov 02 07:29:16.939990 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.939574    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nk4zl" event={"ID":"3c7c9f01-cc9c-405b-ad39-9a4f575074dd","Type":"ContainerDied","Data":"bf1361e47c778fc95c24202849ae4ad456761c3445a5f55b2478c8a094275164"}
Nov 02 07:29:16.939990 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.939703    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf1361e47c778fc95c24202849ae4ad456761c3445a5f55b2478c8a094275164"
Nov 02 07:29:16.984305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.984273    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c4e254a-ffbc-4065-88ce-cb8460dc981a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-x8stq\" (UID: \"8c4e254a-ffbc-4065-88ce-cb8460dc981a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq"
Nov 02 07:29:16.984469 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:16.984328    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rjr9\" (UniqueName: \"kubernetes.io/projected/8c4e254a-ffbc-4065-88ce-cb8460dc981a-kube-api-access-9rjr9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-x8stq\" (UID: \"8c4e254a-ffbc-4065-88ce-cb8460dc981a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq"
Nov 02 07:29:17.085638 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:17.085596    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c4e254a-ffbc-4065-88ce-cb8460dc981a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-x8stq\" (UID: \"8c4e254a-ffbc-4065-88ce-cb8460dc981a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq"
Nov 02 07:29:17.085809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:17.085651    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rjr9\" (UniqueName: \"kubernetes.io/projected/8c4e254a-ffbc-4065-88ce-cb8460dc981a-kube-api-access-9rjr9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-x8stq\" (UID: \"8c4e254a-ffbc-4065-88ce-cb8460dc981a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq"
Nov 02 07:29:17.086049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:17.086026    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c4e254a-ffbc-4065-88ce-cb8460dc981a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-x8stq\" (UID: \"8c4e254a-ffbc-4065-88ce-cb8460dc981a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq"
Nov 02 07:29:17.114476 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:17.114449    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rjr9\" (UniqueName: \"kubernetes.io/projected/8c4e254a-ffbc-4065-88ce-cb8460dc981a-kube-api-access-9rjr9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-x8stq\" (UID: \"8c4e254a-ffbc-4065-88ce-cb8460dc981a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq"
Nov 02 07:29:17.198453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:17.198415    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq"
Nov 02 07:29:17.523305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:17.523227    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq"]
Nov 02 07:29:17.526205 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:29:17.526175    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4e254a_ffbc_4065_88ce_cb8460dc981a.slice/crio-c3dc99f8f60faa1413f62fb90f69bd52a108ea22f2cf43e162e1346ed8e66eae WatchSource:0}: Error finding container c3dc99f8f60faa1413f62fb90f69bd52a108ea22f2cf43e162e1346ed8e66eae: Status 404 returned error can't find the container with id c3dc99f8f60faa1413f62fb90f69bd52a108ea22f2cf43e162e1346ed8e66eae
Nov 02 07:29:17.529252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:17.529233    4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 07:29:17.945174 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:17.945139    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq" event={"ID":"8c4e254a-ffbc-4065-88ce-cb8460dc981a","Type":"ContainerStarted","Data":"c3dc99f8f60faa1413f62fb90f69bd52a108ea22f2cf43e162e1346ed8e66eae"}
Nov 02 07:29:19.956142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:19.956111    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq" event={"ID":"8c4e254a-ffbc-4065-88ce-cb8460dc981a","Type":"ContainerStarted","Data":"a4307bc42c382eac59cad68f55dbcdd0b610e638b761854dd2c9683260783f01"}
Nov 02 07:29:20.024758 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:20.024707    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-x8stq" podStartSLOduration=1.976867604 podStartE2EDuration="4.024692552s" totalImagesPullingTime="2.047824948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:20.012450693 +0000 UTC m=+310.702777678" watchObservedRunningTime="2026-11-02 07:29:20.024692552 +0000 UTC m=+310.715019516"
Nov 02 07:29:25.394518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:25.394436    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5"]
Nov 02 07:29:25.397735 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:25.397715    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5"
Nov 02 07:29:25.427167 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:25.427140    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-95lzh\""
Nov 02 07:29:25.427167 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:25.427141    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 07:29:25.427375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:25.427166    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 07:29:25.428822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:25.428798    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5"]
Nov 02 07:29:25.452693 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:25.452663    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgwqq\" (UniqueName: \"kubernetes.io/projected/a2a4d97a-4060-429a-81e6-dffbfd834a6d-kube-api-access-zgwqq\") pod \"limitador-operator-controller-manager-5778bb4449-pvwn5\" (UID: \"a2a4d97a-4060-429a-81e6-dffbfd834a6d\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5"
Nov 02 07:29:25.553513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:25.553474    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgwqq\" (UniqueName: \"kubernetes.io/projected/a2a4d97a-4060-429a-81e6-dffbfd834a6d-kube-api-access-zgwqq\") pod \"limitador-operator-controller-manager-5778bb4449-pvwn5\" (UID: \"a2a4d97a-4060-429a-81e6-dffbfd834a6d\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5"
Nov 02 07:29:25.577961 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:25.577933    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgwqq\" (UniqueName: \"kubernetes.io/projected/a2a4d97a-4060-429a-81e6-dffbfd834a6d-kube-api-access-zgwqq\") pod \"limitador-operator-controller-manager-5778bb4449-pvwn5\" (UID: \"a2a4d97a-4060-429a-81e6-dffbfd834a6d\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5"
Nov 02 07:29:25.709874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:25.709836    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5"
Nov 02 07:29:26.005852 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:26.005827    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5"]
Nov 02 07:29:26.007804 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:29:26.007772    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a4d97a_4060_429a_81e6_dffbfd834a6d.slice/crio-ad5853c0eaeedd0db4e83dedc79024c4be9d6250b25abe282a272a0fa851f5b1 WatchSource:0}: Error finding container ad5853c0eaeedd0db4e83dedc79024c4be9d6250b25abe282a272a0fa851f5b1: Status 404 returned error can't find the container with id ad5853c0eaeedd0db4e83dedc79024c4be9d6250b25abe282a272a0fa851f5b1
Nov 02 07:29:26.990066 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:26.990019    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5" event={"ID":"a2a4d97a-4060-429a-81e6-dffbfd834a6d","Type":"ContainerStarted","Data":"ad5853c0eaeedd0db4e83dedc79024c4be9d6250b25abe282a272a0fa851f5b1"}
Nov 02 07:29:29.001554 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:29.001516    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5" event={"ID":"a2a4d97a-4060-429a-81e6-dffbfd834a6d","Type":"ContainerStarted","Data":"b71b1504ed3fb4b7fb42e3f970a22bd8fe30633e02509100703c2b6c75568d14"}
Nov 02 07:29:29.001964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:29.001618    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5"
Nov 02 07:29:29.063578 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:29.063526    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5" podStartSLOduration=1.552591565 podStartE2EDuration="4.063512691s" totalImagesPullingTime="2.510921126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:29.062391775 +0000 UTC m=+319.752718758" watchObservedRunningTime="2026-11-02 07:29:29.063512691 +0000 UTC m=+319.753839655"
Nov 02 07:29:30.648643 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:30.648613    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-547hn"]
Nov 02 07:29:30.652031 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:30.652004    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-547hn"
Nov 02 07:29:30.683009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:30.682980    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-547hn"]
Nov 02 07:29:30.698569 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:30.698540    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzq9k\" (UniqueName: \"kubernetes.io/projected/eddc764b-21f2-4cd2-b5b3-d331ad9c131d-kube-api-access-zzq9k\") pod \"authorino-operator-7b7bb4bc77-547hn\" (UID: \"eddc764b-21f2-4cd2-b5b3-d331ad9c131d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-547hn"
Nov 02 07:29:30.717731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:30.717713    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2psln\""
Nov 02 07:29:30.799383 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:30.799350    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzq9k\" (UniqueName: \"kubernetes.io/projected/eddc764b-21f2-4cd2-b5b3-d331ad9c131d-kube-api-access-zzq9k\") pod \"authorino-operator-7b7bb4bc77-547hn\" (UID: \"eddc764b-21f2-4cd2-b5b3-d331ad9c131d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-547hn"
Nov 02 07:29:30.827208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:30.827173    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzq9k\" (UniqueName: \"kubernetes.io/projected/eddc764b-21f2-4cd2-b5b3-d331ad9c131d-kube-api-access-zzq9k\") pod \"authorino-operator-7b7bb4bc77-547hn\" (UID: \"eddc764b-21f2-4cd2-b5b3-d331ad9c131d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-547hn"
Nov 02 07:29:30.964476 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:30.964438    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-547hn"
Nov 02 07:29:31.260139 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:31.260112    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-547hn"]
Nov 02 07:29:31.262058 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:29:31.262033    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddc764b_21f2_4cd2_b5b3_d331ad9c131d.slice/crio-638e07fcc863da568858b53e895804fcdab4437a7d020bb715747aa84fab2ae8 WatchSource:0}: Error finding container 638e07fcc863da568858b53e895804fcdab4437a7d020bb715747aa84fab2ae8: Status 404 returned error can't find the container with id 638e07fcc863da568858b53e895804fcdab4437a7d020bb715747aa84fab2ae8
Nov 02 07:29:32.021790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:32.021750    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-547hn" event={"ID":"eddc764b-21f2-4cd2-b5b3-d331ad9c131d","Type":"ContainerStarted","Data":"638e07fcc863da568858b53e895804fcdab4437a7d020bb715747aa84fab2ae8"}
Nov 02 07:29:33.027857 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:33.027823    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-547hn" event={"ID":"eddc764b-21f2-4cd2-b5b3-d331ad9c131d","Type":"ContainerStarted","Data":"670a7ad9685e5c5b5f52724a37eb5df49b2f65ba4b25075ef108c135fc3d0793"}
Nov 02 07:29:33.028258 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:33.027963    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-547hn"
Nov 02 07:29:33.076162 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:33.076109    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-547hn" podStartSLOduration=1.3871685089999999 podStartE2EDuration="3.076094544s" totalImagesPullingTime="1.688926035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:33.075176119 +0000 UTC m=+323.765503097" watchObservedRunningTime="2026-11-02 07:29:33.076094544 +0000 UTC m=+323.766421507"
Nov 02 07:29:34.180863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:34.180825    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw"]
Nov 02 07:29:34.184418 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:34.184398    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw"
Nov 02 07:29:34.201054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:34.201033    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 07:29:34.213186 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:34.213162    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cqzbd\""
Nov 02 07:29:34.217744 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:34.217720    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw"]
Nov 02 07:29:34.229492 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:34.229467    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82fsq\" (UniqueName: \"kubernetes.io/projected/b78e88ff-b93b-48c6-b466-cac6e05256d9-kube-api-access-82fsq\") pod \"dns-operator-controller-manager-8557cbdb84-nh2gw\" (UID: \"b78e88ff-b93b-48c6-b466-cac6e05256d9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw"
Nov 02 07:29:34.330255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:34.330205    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82fsq\" (UniqueName: \"kubernetes.io/projected/b78e88ff-b93b-48c6-b466-cac6e05256d9-kube-api-access-82fsq\") pod \"dns-operator-controller-manager-8557cbdb84-nh2gw\" (UID: \"b78e88ff-b93b-48c6-b466-cac6e05256d9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw"
Nov 02 07:29:34.353508 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:34.353471    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82fsq\" (UniqueName: \"kubernetes.io/projected/b78e88ff-b93b-48c6-b466-cac6e05256d9-kube-api-access-82fsq\") pod \"dns-operator-controller-manager-8557cbdb84-nh2gw\" (UID: \"b78e88ff-b93b-48c6-b466-cac6e05256d9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw"
Nov 02 07:29:34.497214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:34.497124    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw"
Nov 02 07:29:34.795826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:34.795800    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw"]
Nov 02 07:29:34.797898 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:29:34.797844    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78e88ff_b93b_48c6_b466_cac6e05256d9.slice/crio-d5e868b0b0f10e45f222d67e3f3e460eb93bc6a15773b80d936e13f51c76cc8a WatchSource:0}: Error finding container d5e868b0b0f10e45f222d67e3f3e460eb93bc6a15773b80d936e13f51c76cc8a: Status 404 returned error can't find the container with id d5e868b0b0f10e45f222d67e3f3e460eb93bc6a15773b80d936e13f51c76cc8a
Nov 02 07:29:35.038272 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:35.038232    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw" event={"ID":"b78e88ff-b93b-48c6-b466-cac6e05256d9","Type":"ContainerStarted","Data":"d5e868b0b0f10e45f222d67e3f3e460eb93bc6a15773b80d936e13f51c76cc8a"}
Nov 02 07:29:37.050995 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:37.050903    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw" event={"ID":"b78e88ff-b93b-48c6-b466-cac6e05256d9","Type":"ContainerStarted","Data":"60bfa37b9b729b7728e3c03d7c9382a5d4d8b4ef0047fc209254db848bf0aece"}
Nov 02 07:29:37.050995 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:37.050972    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw"
Nov 02 07:29:37.091797 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:37.091744    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw" podStartSLOduration=1.110602668 podStartE2EDuration="3.091729769s" totalImagesPullingTime="1.981127101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:37.091424165 +0000 UTC m=+327.781751157" watchObservedRunningTime="2026-11-02 07:29:37.091729769 +0000 UTC m=+327.782056732"
Nov 02 07:29:38.759033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.759000    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"]
Nov 02 07:29:38.762837 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.762817    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.781845 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.781812    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 07:29:38.781845 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.781827    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 07:29:38.782062 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.782006    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 07:29:38.793648 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.793623    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jkbxf\""
Nov 02 07:29:38.793768 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.793698    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 07:29:38.793768 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.793704    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 07:29:38.806929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.806906    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"]
Nov 02 07:29:38.868981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.868947    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.868981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.868985    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-tmp\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.869170 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.869010    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddp8k\" (UniqueName: \"kubernetes.io/projected/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-kube-api-access-ddp8k\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.869170 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.869055    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-manifests\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.869170 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.869132    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-metrics-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.869170 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.869158    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-webhook-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.970459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.970433    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddp8k\" (UniqueName: \"kubernetes.io/projected/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-kube-api-access-ddp8k\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.970602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.970469    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-manifests\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.970602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.970496    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-metrics-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.970602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.970525    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-webhook-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.970602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.970595    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.970834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.970628    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-tmp\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.970919 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.970848    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-manifests\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.973197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.973175    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-tmp\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.973318 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.973242    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.973406 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.973383    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-webhook-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:38.973461 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:38.973394    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-metrics-cert\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:39.001781 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:39.001752    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddp8k\" (UniqueName: \"kubernetes.io/projected/3b785e0e-1e6e-4aae-a65a-f6d5508a813a-kube-api-access-ddp8k\") pod \"opendatahub-operator-controller-manager-756f6c987b-mld7r\" (UID: \"3b785e0e-1e6e-4aae-a65a-f6d5508a813a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:39.077951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:39.077859    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:39.434921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:39.434879    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"]
Nov 02 07:29:39.436104 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:29:39.436077    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b785e0e_1e6e_4aae_a65a_f6d5508a813a.slice/crio-272b54f234aff440e8d88139e4b4643fe3c833c5a142c8b7f8409db80a12f8ab WatchSource:0}: Error finding container 272b54f234aff440e8d88139e4b4643fe3c833c5a142c8b7f8409db80a12f8ab: Status 404 returned error can't find the container with id 272b54f234aff440e8d88139e4b4643fe3c833c5a142c8b7f8409db80a12f8ab
Nov 02 07:29:40.008861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:40.008829    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pvwn5"
Nov 02 07:29:40.069068 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:40.069008    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r" event={"ID":"3b785e0e-1e6e-4aae-a65a-f6d5508a813a","Type":"ContainerStarted","Data":"272b54f234aff440e8d88139e4b4643fe3c833c5a142c8b7f8409db80a12f8ab"}
Nov 02 07:29:42.074559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:42.074519    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"]
Nov 02 07:29:42.078446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:42.078420    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"
Nov 02 07:29:42.104739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:42.104709    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hnnz8\""
Nov 02 07:29:42.106966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:42.106943    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"]
Nov 02 07:29:42.203297 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:42.203261    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fsnv\" (UniqueName: \"kubernetes.io/projected/d2c780cc-30a6-43ed-afe5-77b774bb026f-kube-api-access-9fsnv\") pod \"kuadrant-operator-controller-manager-678d87766f-br6sk\" (UID: \"d2c780cc-30a6-43ed-afe5-77b774bb026f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"
Nov 02 07:29:42.203457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:42.203391    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2c780cc-30a6-43ed-afe5-77b774bb026f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-br6sk\" (UID: \"d2c780cc-30a6-43ed-afe5-77b774bb026f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"
Nov 02 07:29:42.304296 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:42.304258    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fsnv\" (UniqueName: \"kubernetes.io/projected/d2c780cc-30a6-43ed-afe5-77b774bb026f-kube-api-access-9fsnv\") pod \"kuadrant-operator-controller-manager-678d87766f-br6sk\" (UID: \"d2c780cc-30a6-43ed-afe5-77b774bb026f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"
Nov 02 07:29:42.304457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:42.304347    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2c780cc-30a6-43ed-afe5-77b774bb026f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-br6sk\" (UID: \"d2c780cc-30a6-43ed-afe5-77b774bb026f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"
Nov 02 07:29:42.304648 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:42.304632    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2c780cc-30a6-43ed-afe5-77b774bb026f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-br6sk\" (UID: \"d2c780cc-30a6-43ed-afe5-77b774bb026f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"
Nov 02 07:29:42.327347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:42.327289    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fsnv\" (UniqueName: \"kubernetes.io/projected/d2c780cc-30a6-43ed-afe5-77b774bb026f-kube-api-access-9fsnv\") pod \"kuadrant-operator-controller-manager-678d87766f-br6sk\" (UID: \"d2c780cc-30a6-43ed-afe5-77b774bb026f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"
Nov 02 07:29:42.394758 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:42.394732    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"
Nov 02 07:29:42.722337 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:42.722313    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"]
Nov 02 07:29:42.724347 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:29:42.724313    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c780cc_30a6_43ed_afe5_77b774bb026f.slice/crio-d740e8f48d51177d8f46dceb32a60e55890871b467b596b9b2077105adb87bb8 WatchSource:0}: Error finding container d740e8f48d51177d8f46dceb32a60e55890871b467b596b9b2077105adb87bb8: Status 404 returned error can't find the container with id d740e8f48d51177d8f46dceb32a60e55890871b467b596b9b2077105adb87bb8
Nov 02 07:29:43.087442 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:43.087355    4898 generic.go:356] "Generic (PLEG): container finished" podID="3b785e0e-1e6e-4aae-a65a-f6d5508a813a" containerID="722cc6ea088b908726ceb59f99732462aba43501b8ad74385143843f27ba1bce" exitCode=0
Nov 02 07:29:43.087864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:43.087434    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r" event={"ID":"3b785e0e-1e6e-4aae-a65a-f6d5508a813a","Type":"ContainerDied","Data":"722cc6ea088b908726ceb59f99732462aba43501b8ad74385143843f27ba1bce"}
Nov 02 07:29:43.088956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:43.088848    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk" event={"ID":"d2c780cc-30a6-43ed-afe5-77b774bb026f","Type":"ContainerStarted","Data":"d740e8f48d51177d8f46dceb32a60e55890871b467b596b9b2077105adb87bb8"}
Nov 02 07:29:44.035153 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:44.035118    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-547hn"
Nov 02 07:29:44.097730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:44.097686    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r" event={"ID":"3b785e0e-1e6e-4aae-a65a-f6d5508a813a","Type":"ContainerStarted","Data":"766ec3d0af2c0c530a563d9226320eda4e0438fb9164f30ce3caa4c5f5e2b4d9"}
Nov 02 07:29:44.098151 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:44.097852    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:44.192545 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:44.192486    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r" podStartSLOduration=2.460477648 podStartE2EDuration="6.192466643s" totalImagesPullingTime="3.200039059s" totalInitContainerRuntime="531.949936ms" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:44.178619231 +0000 UTC m=+334.868946194" watchObservedRunningTime="2026-11-02 07:29:44.192466643 +0000 UTC m=+334.882793608"
Nov 02 07:29:44.992265 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:29:44.992227    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:29:47.121942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:47.121867    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk" event={"ID":"d2c780cc-30a6-43ed-afe5-77b774bb026f","Type":"ContainerStarted","Data":"35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb"}
Nov 02 07:29:47.122379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:47.122004    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"
Nov 02 07:29:47.197560 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:47.197509    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk" podStartSLOduration=1.315713567 podStartE2EDuration="5.197495078s" totalImagesPullingTime="3.881781511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:29:47.196148639 +0000 UTC m=+337.886475619" watchObservedRunningTime="2026-11-02 07:29:47.197495078 +0000 UTC m=+337.887822039"
Nov 02 07:29:48.057349 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:48.057315    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-nh2gw"
Nov 02 07:29:55.105310 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:55.105275    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-756f6c987b-mld7r"
Nov 02 07:29:58.128641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:29:58.128609    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"
Nov 02 07:30:07.282820 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.282787    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"]
Nov 02 07:30:07.289778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.289756    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.307176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.307146    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wpn7p\""
Nov 02 07:30:07.320855 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.320829    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"]
Nov 02 07:30:07.425932 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.425882    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.426090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.425952    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.426090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.426016    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.426090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.426039    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.426090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.426058    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.426229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.426205    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.426262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.426240    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.426309 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.426295    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.426346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.426318    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsvbh\" (UniqueName: \"kubernetes.io/projected/afd78003-f5b8-44eb-91bd-ea259bd3ee02-kube-api-access-qsvbh\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.527307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.527272    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.527307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.527308    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.527505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.527337    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.527505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.527363    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsvbh\" (UniqueName: \"kubernetes.io/projected/afd78003-f5b8-44eb-91bd-ea259bd3ee02-kube-api-access-qsvbh\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.527505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.527410    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.527505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.527445    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.527505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.527472    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.527505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.527492    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.527810 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.527519    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.527810 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.527746    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.527940 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.527807    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.528004 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.527955    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.528056 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.528023    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.528275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.528250    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.529751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.529726    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.530051 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.530034    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.561918 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.561844    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd78003-f5b8-44eb-91bd-ea259bd3ee02-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.562009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.561998    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsvbh\" (UniqueName: \"kubernetes.io/projected/afd78003-f5b8-44eb-91bd-ea259bd3ee02-kube-api-access-qsvbh\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb\" (UID: \"afd78003-f5b8-44eb-91bd-ea259bd3ee02\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.607974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.607948    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:07.920502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.920473    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"]
Nov 02 07:30:07.922936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:30:07.922901    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd78003_f5b8_44eb_91bd_ea259bd3ee02.slice/crio-642a587924b32e6d4bc5dacba0c494e37dcf1b99a36f8dd5b1043b444e23f065 WatchSource:0}: Error finding container 642a587924b32e6d4bc5dacba0c494e37dcf1b99a36f8dd5b1043b444e23f065: Status 404 returned error can't find the container with id 642a587924b32e6d4bc5dacba0c494e37dcf1b99a36f8dd5b1043b444e23f065
Nov 02 07:30:07.925759 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.925710    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:30:07.925851 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.925807    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:30:07.925932 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:07.925851    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:30:08.224159 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:08.224069    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb" event={"ID":"afd78003-f5b8-44eb-91bd-ea259bd3ee02","Type":"ContainerStarted","Data":"890afa865e5c3164f029b941791d8497fbef69a8b84d037d16b7c020a5f03b52"}
Nov 02 07:30:08.224159 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:08.224138    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb" event={"ID":"afd78003-f5b8-44eb-91bd-ea259bd3ee02","Type":"ContainerStarted","Data":"642a587924b32e6d4bc5dacba0c494e37dcf1b99a36f8dd5b1043b444e23f065"}
Nov 02 07:30:08.270265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:08.270219    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb" podStartSLOduration=1.27020505 podStartE2EDuration="1.27020505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:30:08.268187686 +0000 UTC m=+358.958514650" watchObservedRunningTime="2026-11-02 07:30:08.27020505 +0000 UTC m=+358.960532014"
Nov 02 07:30:08.608638 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:08.608564    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:08.613770 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:08.613746    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:09.229855 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:09.229821    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:09.230791 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:09.230772    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb"
Nov 02 07:30:09.478504 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:09.478475    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-754zx"]
Nov 02 07:30:09.481958 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:09.481899    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-754zx"
Nov 02 07:30:09.497782 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:09.497755    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-754zx"]
Nov 02 07:30:09.501753 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:09.501732    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kpq4g\""
Nov 02 07:30:09.646351 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:09.646312    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75wmf\" (UniqueName: \"kubernetes.io/projected/a4c0174d-35a4-4d60-8f26-5cbecefb73dc-kube-api-access-75wmf\") pod \"authorino-57f7f8659f-754zx\" (UID: \"a4c0174d-35a4-4d60-8f26-5cbecefb73dc\") " pod="kuadrant-system/authorino-57f7f8659f-754zx"
Nov 02 07:30:09.747126 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:09.747047    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75wmf\" (UniqueName: \"kubernetes.io/projected/a4c0174d-35a4-4d60-8f26-5cbecefb73dc-kube-api-access-75wmf\") pod \"authorino-57f7f8659f-754zx\" (UID: \"a4c0174d-35a4-4d60-8f26-5cbecefb73dc\") " pod="kuadrant-system/authorino-57f7f8659f-754zx"
Nov 02 07:30:09.770391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:09.770362    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75wmf\" (UniqueName: \"kubernetes.io/projected/a4c0174d-35a4-4d60-8f26-5cbecefb73dc-kube-api-access-75wmf\") pod \"authorino-57f7f8659f-754zx\" (UID: \"a4c0174d-35a4-4d60-8f26-5cbecefb73dc\") " pod="kuadrant-system/authorino-57f7f8659f-754zx"
Nov 02 07:30:09.805213 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:09.805177    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-754zx"
Nov 02 07:30:10.035402 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.035329    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"]
Nov 02 07:30:10.037968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.037951    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.049097 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.049075    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8f5226-3ccd-438c-91b7-253db3b91c4e-metrics-certs\") pod \"jobset-controller-manager-654dd4f8f5-7d2pz\" (UID: \"8a8f5226-3ccd-438c-91b7-253db3b91c4e\") " pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.049214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.049119    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-999lq\" (UniqueName: \"kubernetes.io/projected/8a8f5226-3ccd-438c-91b7-253db3b91c4e-kube-api-access-999lq\") pod \"jobset-controller-manager-654dd4f8f5-7d2pz\" (UID: \"8a8f5226-3ccd-438c-91b7-253db3b91c4e\") " pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.049214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.049196    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a8f5226-3ccd-438c-91b7-253db3b91c4e-manager-config\") pod \"jobset-controller-manager-654dd4f8f5-7d2pz\" (UID: \"8a8f5226-3ccd-438c-91b7-253db3b91c4e\") " pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.049286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.049268    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a8f5226-3ccd-438c-91b7-253db3b91c4e-cert\") pod \"jobset-controller-manager-654dd4f8f5-7d2pz\" (UID: \"8a8f5226-3ccd-438c-91b7-253db3b91c4e\") " pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.059468 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.059434    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 07:30:10.060487 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.060468    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"]
Nov 02 07:30:10.067162 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.067143    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-k8cdh\""
Nov 02 07:30:10.067356 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.067344    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 07:30:10.096109 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.096088    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-754zx"]
Nov 02 07:30:10.097807 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:30:10.097785    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4c0174d_35a4_4d60_8f26_5cbecefb73dc.slice/crio-e2e5f837be25b8a35e65e3c360ef4c2bfa78bbbf6824ea0fd1300ee879a82c11 WatchSource:0}: Error finding container e2e5f837be25b8a35e65e3c360ef4c2bfa78bbbf6824ea0fd1300ee879a82c11: Status 404 returned error can't find the container with id e2e5f837be25b8a35e65e3c360ef4c2bfa78bbbf6824ea0fd1300ee879a82c11
Nov 02 07:30:10.141257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.141233    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 07:30:10.141360 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.141313    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 07:30:10.141360 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.141314    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 07:30:10.149822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.149795    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8f5226-3ccd-438c-91b7-253db3b91c4e-metrics-certs\") pod \"jobset-controller-manager-654dd4f8f5-7d2pz\" (UID: \"8a8f5226-3ccd-438c-91b7-253db3b91c4e\") " pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.149933 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.149834    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-999lq\" (UniqueName: \"kubernetes.io/projected/8a8f5226-3ccd-438c-91b7-253db3b91c4e-kube-api-access-999lq\") pod \"jobset-controller-manager-654dd4f8f5-7d2pz\" (UID: \"8a8f5226-3ccd-438c-91b7-253db3b91c4e\") " pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.149933 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.149865    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a8f5226-3ccd-438c-91b7-253db3b91c4e-manager-config\") pod \"jobset-controller-manager-654dd4f8f5-7d2pz\" (UID: \"8a8f5226-3ccd-438c-91b7-253db3b91c4e\") " pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.149933 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.149911    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a8f5226-3ccd-438c-91b7-253db3b91c4e-cert\") pod \"jobset-controller-manager-654dd4f8f5-7d2pz\" (UID: \"8a8f5226-3ccd-438c-91b7-253db3b91c4e\") " pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.150594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.150572    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a8f5226-3ccd-438c-91b7-253db3b91c4e-manager-config\") pod \"jobset-controller-manager-654dd4f8f5-7d2pz\" (UID: \"8a8f5226-3ccd-438c-91b7-253db3b91c4e\") " pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.152380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.152361    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8f5226-3ccd-438c-91b7-253db3b91c4e-metrics-certs\") pod \"jobset-controller-manager-654dd4f8f5-7d2pz\" (UID: \"8a8f5226-3ccd-438c-91b7-253db3b91c4e\") " pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.152463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.152419    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a8f5226-3ccd-438c-91b7-253db3b91c4e-cert\") pod \"jobset-controller-manager-654dd4f8f5-7d2pz\" (UID: \"8a8f5226-3ccd-438c-91b7-253db3b91c4e\") " pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.172743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.172722    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-999lq\" (UniqueName: \"kubernetes.io/projected/8a8f5226-3ccd-438c-91b7-253db3b91c4e-kube-api-access-999lq\") pod \"jobset-controller-manager-654dd4f8f5-7d2pz\" (UID: \"8a8f5226-3ccd-438c-91b7-253db3b91c4e\") " pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.236285 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.236250    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-754zx" event={"ID":"a4c0174d-35a4-4d60-8f26-5cbecefb73dc","Type":"ContainerStarted","Data":"e2e5f837be25b8a35e65e3c360ef4c2bfa78bbbf6824ea0fd1300ee879a82c11"}
Nov 02 07:30:10.349256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.349197    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:10.643925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:10.643899    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"]
Nov 02 07:30:10.645467 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:30:10.645439    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8f5226_3ccd_438c_91b7_253db3b91c4e.slice/crio-f878eb0c0820b5c86f938e28bfe2d82d14ae7376e59732f3d55d22d354e693fb WatchSource:0}: Error finding container f878eb0c0820b5c86f938e28bfe2d82d14ae7376e59732f3d55d22d354e693fb: Status 404 returned error can't find the container with id f878eb0c0820b5c86f938e28bfe2d82d14ae7376e59732f3d55d22d354e693fb
Nov 02 07:30:11.243253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:11.243209    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz" event={"ID":"8a8f5226-3ccd-438c-91b7-253db3b91c4e","Type":"ContainerStarted","Data":"f878eb0c0820b5c86f938e28bfe2d82d14ae7376e59732f3d55d22d354e693fb"}
Nov 02 07:30:14.264998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:14.264953    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz" event={"ID":"8a8f5226-3ccd-438c-91b7-253db3b91c4e","Type":"ContainerStarted","Data":"8770025d564748070f06c77bbea90c1351b262d4ef0b02dea6f7aba1bae2fb34"}
Nov 02 07:30:14.265531 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:14.265142    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:14.266473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:14.266445    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-754zx" event={"ID":"a4c0174d-35a4-4d60-8f26-5cbecefb73dc","Type":"ContainerStarted","Data":"b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038"}
Nov 02 07:30:14.313102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:14.313042    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz" podStartSLOduration=1.072015752 podStartE2EDuration="4.313027945s" totalImagesPullingTime="3.241012193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:30:14.311153805 +0000 UTC m=+365.001480786" watchObservedRunningTime="2026-11-02 07:30:14.313027945 +0000 UTC m=+365.003354909"
Nov 02 07:30:14.350909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:14.350833    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-754zx" podStartSLOduration=1.675208985 podStartE2EDuration="5.35081772s" totalImagesPullingTime="3.675608735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:30:14.350349948 +0000 UTC m=+365.040677028" watchObservedRunningTime="2026-11-02 07:30:14.35081772 +0000 UTC m=+365.041144684"
Nov 02 07:30:14.988329 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:30:14.988292    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:30:15.262021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:15.261941    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-754zx"]
Nov 02 07:30:16.276328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:16.276283    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-754zx" podUID="a4c0174d-35a4-4d60-8f26-5cbecefb73dc" containerName="authorino" containerID="cri-o://b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038" gracePeriod=30
Nov 02 07:30:16.525751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:16.525723    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-754zx"
Nov 02 07:30:16.608525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:16.608457    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c0174d-35a4-4d60-8f26-5cbecefb73dc-kube-api-access-75wmf\" (UniqueName: \"kubernetes.io/projected/a4c0174d-35a4-4d60-8f26-5cbecefb73dc-kube-api-access-75wmf\") pod \"a4c0174d-35a4-4d60-8f26-5cbecefb73dc\" (UID: \"a4c0174d-35a4-4d60-8f26-5cbecefb73dc\") "
Nov 02 07:30:16.610574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:16.610549    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c0174d-35a4-4d60-8f26-5cbecefb73dc-kube-api-access-75wmf" pod "a4c0174d-35a4-4d60-8f26-5cbecefb73dc" (UID: "a4c0174d-35a4-4d60-8f26-5cbecefb73dc"). InnerVolumeSpecName "kube-api-access-75wmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:30:16.709152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:16.709120    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75wmf\" (UniqueName: \"kubernetes.io/projected/a4c0174d-35a4-4d60-8f26-5cbecefb73dc-kube-api-access-75wmf\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:30:17.282455 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:17.282422    4898 generic.go:356] "Generic (PLEG): container finished" podID="a4c0174d-35a4-4d60-8f26-5cbecefb73dc" containerID="b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038" exitCode=0
Nov 02 07:30:17.282860 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:17.282469    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-754zx"
Nov 02 07:30:17.282860 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:17.282511    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-754zx" event={"ID":"a4c0174d-35a4-4d60-8f26-5cbecefb73dc","Type":"ContainerDied","Data":"b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038"}
Nov 02 07:30:17.282860 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:17.282555    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-754zx" event={"ID":"a4c0174d-35a4-4d60-8f26-5cbecefb73dc","Type":"ContainerDied","Data":"e2e5f837be25b8a35e65e3c360ef4c2bfa78bbbf6824ea0fd1300ee879a82c11"}
Nov 02 07:30:17.282860 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:17.282578    4898 scope.go:122] "RemoveContainer" containerID="b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038"
Nov 02 07:30:17.291689 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:17.291673    4898 scope.go:122] "RemoveContainer" containerID="b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038"
Nov 02 07:30:17.291943 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:30:17.291919    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038\": container with ID starting with b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038 not found: ID does not exist" containerID="b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038"
Nov 02 07:30:17.292054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:17.291950    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038"} err="failed to get container status \"b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038\": rpc error: code = NotFound desc = could not find container \"b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038\": container with ID starting with b5c642750077b942c52d78f70b9472a3f222602c82578d03581611e74fb4c038 not found: ID does not exist"
Nov 02 07:30:17.331266 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:17.331241    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-754zx"]
Nov 02 07:30:17.333325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:17.333306    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-754zx"]
Nov 02 07:30:17.462205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:17.462173    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c0174d-35a4-4d60-8f26-5cbecefb73dc" path="/var/lib/kubelet/pods/a4c0174d-35a4-4d60-8f26-5cbecefb73dc/volumes"
Nov 02 07:30:25.277151 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:30:25.277120    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-654dd4f8f5-7d2pz"
Nov 02 07:30:44.986968 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:30:44.986926    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:31:10.390369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.390335    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-rcnpx"]
Nov 02 07:31:10.390916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.390885    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a4c0174d-35a4-4d60-8f26-5cbecefb73dc" containerName="authorino"
Nov 02 07:31:10.390996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.390922    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c0174d-35a4-4d60-8f26-5cbecefb73dc" containerName="authorino"
Nov 02 07:31:10.391049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.391037    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c0174d-35a4-4d60-8f26-5cbecefb73dc" containerName="authorino"
Nov 02 07:31:10.395287 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.395265    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:10.424085 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.424049    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 07:31:10.424085 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.424052    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 07:31:10.429653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.428031    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-rcnpx"]
Nov 02 07:31:10.433710 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.433689    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 02 07:31:10.433710 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.433705    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ww4lh\""
Nov 02 07:31:10.433900 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.433818    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 02 07:31:10.461483 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.461445    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f47vx\" (UniqueName: \"kubernetes.io/projected/7c663e0a-d1b2-4ea4-91d3-fe88764f387d-kube-api-access-f47vx\") pod \"odh-model-controller-769875db56-rcnpx\" (UID: \"7c663e0a-d1b2-4ea4-91d3-fe88764f387d\") " pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:10.461620 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.461519    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c663e0a-d1b2-4ea4-91d3-fe88764f387d-metrics-certs\") pod \"odh-model-controller-769875db56-rcnpx\" (UID: \"7c663e0a-d1b2-4ea4-91d3-fe88764f387d\") " pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:10.461620 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.461599    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c663e0a-d1b2-4ea4-91d3-fe88764f387d-cert\") pod \"odh-model-controller-769875db56-rcnpx\" (UID: \"7c663e0a-d1b2-4ea4-91d3-fe88764f387d\") " pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:10.562084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.562050    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c663e0a-d1b2-4ea4-91d3-fe88764f387d-cert\") pod \"odh-model-controller-769875db56-rcnpx\" (UID: \"7c663e0a-d1b2-4ea4-91d3-fe88764f387d\") " pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:10.562278 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.562128    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f47vx\" (UniqueName: \"kubernetes.io/projected/7c663e0a-d1b2-4ea4-91d3-fe88764f387d-kube-api-access-f47vx\") pod \"odh-model-controller-769875db56-rcnpx\" (UID: \"7c663e0a-d1b2-4ea4-91d3-fe88764f387d\") " pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:10.562278 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.562166    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c663e0a-d1b2-4ea4-91d3-fe88764f387d-metrics-certs\") pod \"odh-model-controller-769875db56-rcnpx\" (UID: \"7c663e0a-d1b2-4ea4-91d3-fe88764f387d\") " pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:10.564554 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.564528    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c663e0a-d1b2-4ea4-91d3-fe88764f387d-cert\") pod \"odh-model-controller-769875db56-rcnpx\" (UID: \"7c663e0a-d1b2-4ea4-91d3-fe88764f387d\") " pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:10.564691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.564633    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c663e0a-d1b2-4ea4-91d3-fe88764f387d-metrics-certs\") pod \"odh-model-controller-769875db56-rcnpx\" (UID: \"7c663e0a-d1b2-4ea4-91d3-fe88764f387d\") " pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:10.585175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.585150    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f47vx\" (UniqueName: \"kubernetes.io/projected/7c663e0a-d1b2-4ea4-91d3-fe88764f387d-kube-api-access-f47vx\") pod \"odh-model-controller-769875db56-rcnpx\" (UID: \"7c663e0a-d1b2-4ea4-91d3-fe88764f387d\") " pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:10.708000 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:10.707966    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:11.003054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:11.003028    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-rcnpx"]
Nov 02 07:31:11.003318 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:31:11.003296    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c663e0a_d1b2_4ea4_91d3_fe88764f387d.slice/crio-d8b1eb1ddfde104153d3114062ef7694397ea5134a2b2bccb7188cac7e058051 WatchSource:0}: Error finding container d8b1eb1ddfde104153d3114062ef7694397ea5134a2b2bccb7188cac7e058051: Status 404 returned error can't find the container with id d8b1eb1ddfde104153d3114062ef7694397ea5134a2b2bccb7188cac7e058051
Nov 02 07:31:11.546045 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:11.546006    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-rcnpx" event={"ID":"7c663e0a-d1b2-4ea4-91d3-fe88764f387d","Type":"ContainerStarted","Data":"d8b1eb1ddfde104153d3114062ef7694397ea5134a2b2bccb7188cac7e058051"}
Nov 02 07:31:14.564516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:14.564473    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-rcnpx" event={"ID":"7c663e0a-d1b2-4ea4-91d3-fe88764f387d","Type":"ContainerStarted","Data":"9cac13ff4047dc85d786e5be651d0f82568bf64cf12af23a2cd93cad4b2cf4be"}
Nov 02 07:31:14.564975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:14.564544    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:14.606819 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:14.606766    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-rcnpx" podStartSLOduration=1.841243746 podStartE2EDuration="4.60675206s" totalImagesPullingTime="2.765508314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:31:14.605637278 +0000 UTC m=+425.295964243" watchObservedRunningTime="2026-11-02 07:31:14.60675206 +0000 UTC m=+425.297079024"
Nov 02 07:31:14.987813 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:31:14.987773    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:31:25.571625 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:25.571597    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-rcnpx"
Nov 02 07:31:35.869054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:35.869014    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jjwgc"]
Nov 02 07:31:35.874721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:35.874695    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jjwgc"
Nov 02 07:31:35.892926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:35.892874    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jjwgc"]
Nov 02 07:31:35.903048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:35.903027    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kpq4g\""
Nov 02 07:31:35.998257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:35.998223    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bz9\" (UniqueName: \"kubernetes.io/projected/bc1cae85-d84b-4347-a833-d0223151baa8-kube-api-access-f6bz9\") pod \"authorino-6f6944bcdd-jjwgc\" (UID: \"bc1cae85-d84b-4347-a833-d0223151baa8\") " pod="kuadrant-system/authorino-6f6944bcdd-jjwgc"
Nov 02 07:31:36.080834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.080801    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jjwgc"]
Nov 02 07:31:36.081041 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:31:36.081022    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-f6bz9], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-jjwgc" podUID="bc1cae85-d84b-4347-a833-d0223151baa8"
Nov 02 07:31:36.099426 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.099384    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bz9\" (UniqueName: \"kubernetes.io/projected/bc1cae85-d84b-4347-a833-d0223151baa8-kube-api-access-f6bz9\") pod \"authorino-6f6944bcdd-jjwgc\" (UID: \"bc1cae85-d84b-4347-a833-d0223151baa8\") " pod="kuadrant-system/authorino-6f6944bcdd-jjwgc"
Nov 02 07:31:36.108373 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.108343    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hrfwq"]
Nov 02 07:31:36.111669 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.111646    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hrfwq"
Nov 02 07:31:36.123118 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.123070    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bz9\" (UniqueName: \"kubernetes.io/projected/bc1cae85-d84b-4347-a833-d0223151baa8-kube-api-access-f6bz9\") pod \"authorino-6f6944bcdd-jjwgc\" (UID: \"bc1cae85-d84b-4347-a833-d0223151baa8\") " pod="kuadrant-system/authorino-6f6944bcdd-jjwgc"
Nov 02 07:31:36.132305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.132282    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hrfwq"]
Nov 02 07:31:36.189686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.189655    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 07:31:36.200810 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.200781    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9l9l\" (UniqueName: \"kubernetes.io/projected/086e0ea0-bd50-480e-ade3-348b250bb07e-kube-api-access-x9l9l\") pod \"authorino-6889cbd8c4-hrfwq\" (UID: \"086e0ea0-bd50-480e-ade3-348b250bb07e\") " pod="kuadrant-system/authorino-6889cbd8c4-hrfwq"
Nov 02 07:31:36.201027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.201000    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/086e0ea0-bd50-480e-ade3-348b250bb07e-tls-cert\") pod \"authorino-6889cbd8c4-hrfwq\" (UID: \"086e0ea0-bd50-480e-ade3-348b250bb07e\") " pod="kuadrant-system/authorino-6889cbd8c4-hrfwq"
Nov 02 07:31:36.229726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.229699    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hrfwq"]
Nov 02 07:31:36.229919 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:31:36.229901    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-x9l9l tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-hrfwq" podUID="086e0ea0-bd50-480e-ade3-348b250bb07e"
Nov 02 07:31:36.257495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.257469    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7564847bc7-zr86m"]
Nov 02 07:31:36.260572 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.260556    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7564847bc7-zr86m"
Nov 02 07:31:36.280871 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.280847    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7564847bc7-zr86m"]
Nov 02 07:31:36.301418 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.301395    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9l9l\" (UniqueName: \"kubernetes.io/projected/086e0ea0-bd50-480e-ade3-348b250bb07e-kube-api-access-x9l9l\") pod \"authorino-6889cbd8c4-hrfwq\" (UID: \"086e0ea0-bd50-480e-ade3-348b250bb07e\") " pod="kuadrant-system/authorino-6889cbd8c4-hrfwq"
Nov 02 07:31:36.301540 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.301509    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/086e0ea0-bd50-480e-ade3-348b250bb07e-tls-cert\") pod \"authorino-6889cbd8c4-hrfwq\" (UID: \"086e0ea0-bd50-480e-ade3-348b250bb07e\") " pod="kuadrant-system/authorino-6889cbd8c4-hrfwq"
Nov 02 07:31:36.304065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.304040    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/086e0ea0-bd50-480e-ade3-348b250bb07e-tls-cert\") pod \"authorino-6889cbd8c4-hrfwq\" (UID: \"086e0ea0-bd50-480e-ade3-348b250bb07e\") " pod="kuadrant-system/authorino-6889cbd8c4-hrfwq"
Nov 02 07:31:36.325027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.325003    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9l9l\" (UniqueName: \"kubernetes.io/projected/086e0ea0-bd50-480e-ade3-348b250bb07e-kube-api-access-x9l9l\") pod \"authorino-6889cbd8c4-hrfwq\" (UID: \"086e0ea0-bd50-480e-ade3-348b250bb07e\") " pod="kuadrant-system/authorino-6889cbd8c4-hrfwq"
Nov 02 07:31:36.402153 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.402128    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-tls-cert\") pod \"authorino-7564847bc7-zr86m\" (UID: \"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51\") " pod="kuadrant-system/authorino-7564847bc7-zr86m"
Nov 02 07:31:36.402264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.402174    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lflwk\" (UniqueName: \"kubernetes.io/projected/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-kube-api-access-lflwk\") pod \"authorino-7564847bc7-zr86m\" (UID: \"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51\") " pod="kuadrant-system/authorino-7564847bc7-zr86m"
Nov 02 07:31:36.503691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.503645    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-tls-cert\") pod \"authorino-7564847bc7-zr86m\" (UID: \"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51\") " pod="kuadrant-system/authorino-7564847bc7-zr86m"
Nov 02 07:31:36.503846 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.503746    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lflwk\" (UniqueName: \"kubernetes.io/projected/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-kube-api-access-lflwk\") pod \"authorino-7564847bc7-zr86m\" (UID: \"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51\") " pod="kuadrant-system/authorino-7564847bc7-zr86m"
Nov 02 07:31:36.506195 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.506173    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-tls-cert\") pod \"authorino-7564847bc7-zr86m\" (UID: \"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51\") " pod="kuadrant-system/authorino-7564847bc7-zr86m"
Nov 02 07:31:36.527461 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.527432    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lflwk\" (UniqueName: \"kubernetes.io/projected/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-kube-api-access-lflwk\") pod \"authorino-7564847bc7-zr86m\" (UID: \"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51\") " pod="kuadrant-system/authorino-7564847bc7-zr86m"
Nov 02 07:31:36.589388 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.589357    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7564847bc7-zr86m"
Nov 02 07:31:36.703392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.703352    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hrfwq"
Nov 02 07:31:36.703561 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.703352    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jjwgc"
Nov 02 07:31:36.720612 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.720580    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hrfwq"
Nov 02 07:31:36.723621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.723594    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jjwgc"
Nov 02 07:31:36.806571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.806538    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086e0ea0-bd50-480e-ade3-348b250bb07e-kube-api-access-x9l9l\" (UniqueName: \"kubernetes.io/projected/086e0ea0-bd50-480e-ade3-348b250bb07e-kube-api-access-x9l9l\") pod \"086e0ea0-bd50-480e-ade3-348b250bb07e\" (UID: \"086e0ea0-bd50-480e-ade3-348b250bb07e\") "
Nov 02 07:31:36.806757 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.806588    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/086e0ea0-bd50-480e-ade3-348b250bb07e-tls-cert\" (UniqueName: \"kubernetes.io/secret/086e0ea0-bd50-480e-ade3-348b250bb07e-tls-cert\") pod \"086e0ea0-bd50-480e-ade3-348b250bb07e\" (UID: \"086e0ea0-bd50-480e-ade3-348b250bb07e\") "
Nov 02 07:31:36.806757 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.806681    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1cae85-d84b-4347-a833-d0223151baa8-kube-api-access-f6bz9\" (UniqueName: \"kubernetes.io/projected/bc1cae85-d84b-4347-a833-d0223151baa8-kube-api-access-f6bz9\") pod \"bc1cae85-d84b-4347-a833-d0223151baa8\" (UID: \"bc1cae85-d84b-4347-a833-d0223151baa8\") "
Nov 02 07:31:36.808853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.808824    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086e0ea0-bd50-480e-ade3-348b250bb07e-kube-api-access-x9l9l" pod "086e0ea0-bd50-480e-ade3-348b250bb07e" (UID: "086e0ea0-bd50-480e-ade3-348b250bb07e"). InnerVolumeSpecName "kube-api-access-x9l9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:31:36.808853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.808826    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086e0ea0-bd50-480e-ade3-348b250bb07e-tls-cert" pod "086e0ea0-bd50-480e-ade3-348b250bb07e" (UID: "086e0ea0-bd50-480e-ade3-348b250bb07e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:31:36.809020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.808925    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1cae85-d84b-4347-a833-d0223151baa8-kube-api-access-f6bz9" pod "bc1cae85-d84b-4347-a833-d0223151baa8" (UID: "bc1cae85-d84b-4347-a833-d0223151baa8"). InnerVolumeSpecName "kube-api-access-f6bz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:31:36.893013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.892988    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7564847bc7-zr86m"]
Nov 02 07:31:36.893555 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:31:36.893533    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod284c9e7f_d9e9_4ae7_8d8e_4390045d2f51.slice/crio-1f3c2996b5421ff7ffa0da2eb808dfbf71f236768d6c62d10e8bab88b0664ced WatchSource:0}: Error finding container 1f3c2996b5421ff7ffa0da2eb808dfbf71f236768d6c62d10e8bab88b0664ced: Status 404 returned error can't find the container with id 1f3c2996b5421ff7ffa0da2eb808dfbf71f236768d6c62d10e8bab88b0664ced
Nov 02 07:31:36.907881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.907828    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9l9l\" (UniqueName: \"kubernetes.io/projected/086e0ea0-bd50-480e-ade3-348b250bb07e-kube-api-access-x9l9l\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:31:36.907881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.907851    4898 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/086e0ea0-bd50-480e-ade3-348b250bb07e-tls-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:31:36.907881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:36.907863    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6bz9\" (UniqueName: \"kubernetes.io/projected/bc1cae85-d84b-4347-a833-d0223151baa8-kube-api-access-f6bz9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:31:37.710409 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.710372    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7564847bc7-zr86m" event={"ID":"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51","Type":"ContainerStarted","Data":"50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51"}
Nov 02 07:31:37.710409 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.710419    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7564847bc7-zr86m" event={"ID":"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51","Type":"ContainerStarted","Data":"1f3c2996b5421ff7ffa0da2eb808dfbf71f236768d6c62d10e8bab88b0664ced"}
Nov 02 07:31:37.711976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.711951    4898 generic.go:356] "Generic (PLEG): container finished" podID="dea9d7e7-103a-4e9a-8824-e62a72191cac" containerID="77a4f343749c867c13f39ed4e98cecfce7f13babf762ba275a91acb1fe3926b8" exitCode=0
Nov 02 07:31:37.712119 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.712063    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jjwgc"
Nov 02 07:31:37.712119 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.712095    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" event={"ID":"dea9d7e7-103a-4e9a-8824-e62a72191cac","Type":"ContainerDied","Data":"77a4f343749c867c13f39ed4e98cecfce7f13babf762ba275a91acb1fe3926b8"}
Nov 02 07:31:37.712225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.712140    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hrfwq"
Nov 02 07:31:37.712641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.712620    4898 scope.go:122] "RemoveContainer" containerID="77a4f343749c867c13f39ed4e98cecfce7f13babf762ba275a91acb1fe3926b8"
Nov 02 07:31:37.713834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.713804    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:31:37.714054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.713935    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:31:37.714054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.713984    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:31:37.751717 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.751670    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7564847bc7-zr86m" podStartSLOduration=1.113204664 podStartE2EDuration="1.751656671s" totalImagesPullingTime="638.452007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:31:37.749951115 +0000 UTC m=+448.440278082" watchObservedRunningTime="2026-11-02 07:31:37.751656671 +0000 UTC m=+448.441983634"
Nov 02 07:31:37.812961 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.812926    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hrfwq"]
Nov 02 07:31:37.821148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.821119    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hrfwq"]
Nov 02 07:31:37.929861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.929826    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jjwgc"]
Nov 02 07:31:37.937118 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:37.937078    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jjwgc"]
Nov 02 07:31:38.719025 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:38.718986    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" event={"ID":"dea9d7e7-103a-4e9a-8824-e62a72191cac","Type":"ContainerStarted","Data":"d9ca1d3094ba6cb6aab56c7febcfc0dd9c8813f04a44332c124414fdddda04fc"}
Nov 02 07:31:38.976972 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:38.976272    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:31:38.979911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:38.979560    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 07:31:38.979911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:38.979630    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" podUID="dea9d7e7-103a-4e9a-8824-e62a72191cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 07:31:39.486249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:39.486217    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086e0ea0-bd50-480e-ade3-348b250bb07e" path="/var/lib/kubelet/pods/086e0ea0-bd50-480e-ade3-348b250bb07e/volumes"
Nov 02 07:31:39.486472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:39.486460    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1cae85-d84b-4347-a833-d0223151baa8" path="/var/lib/kubelet/pods/bc1cae85-d84b-4347-a833-d0223151baa8/volumes"
Nov 02 07:31:39.979916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:39.978280    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 07:31:39.979916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:39.978340    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" podUID="dea9d7e7-103a-4e9a-8824-e62a72191cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 07:31:40.980313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:40.980278    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 07:31:40.980779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:40.980746    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" podUID="dea9d7e7-103a-4e9a-8824-e62a72191cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 07:31:41.980031 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:41.979990    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 07:31:41.980273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:41.980246    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" podUID="dea9d7e7-103a-4e9a-8824-e62a72191cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 07:31:42.731358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:42.731325    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:31:42.976299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:42.976264    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 02 07:31:42.976577 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:42.976548    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" podUID="dea9d7e7-103a-4e9a-8824-e62a72191cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 02 07:31:43.979357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:43.979325    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:31:43.980507 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:31:43.980485    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t"
Nov 02 07:31:44.987598 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:31:44.987559    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:32:14.987282 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:32:14.987247    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:32:44.988613 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:32:44.988572    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:33:14.987516 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:33:14.987471    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:33:32.861441 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:32.861399    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"]
Nov 02 07:33:32.865403 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:32.865383    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:32.903412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:32.903389    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 07:33:32.903678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:32.903655    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 07:33:32.903770 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:32.903662    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 07:33:32.903770 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:32.903736    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dj6tl\""
Nov 02 07:33:32.905258 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:32.905237    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"]
Nov 02 07:33:33.003577 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.003544    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545d9172-81e3-409a-ae24-163f0646b018-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.003752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.003591    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545d9172-81e3-409a-ae24-163f0646b018-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.003752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.003613    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545d9172-81e3-409a-ae24-163f0646b018-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.003752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.003683    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545d9172-81e3-409a-ae24-163f0646b018-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.003752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.003735    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd5mb\" (UniqueName: \"kubernetes.io/projected/545d9172-81e3-409a-ae24-163f0646b018-kube-api-access-nd5mb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.003942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.003760    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545d9172-81e3-409a-ae24-163f0646b018-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.104305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.104273    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545d9172-81e3-409a-ae24-163f0646b018-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.104493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.104326    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545d9172-81e3-409a-ae24-163f0646b018-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.104493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.104345    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545d9172-81e3-409a-ae24-163f0646b018-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.104493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.104464    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545d9172-81e3-409a-ae24-163f0646b018-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.104663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.104530    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd5mb\" (UniqueName: \"kubernetes.io/projected/545d9172-81e3-409a-ae24-163f0646b018-kube-api-access-nd5mb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.104663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.104574    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545d9172-81e3-409a-ae24-163f0646b018-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.104765 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.104704    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545d9172-81e3-409a-ae24-163f0646b018-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.104765 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.104725    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545d9172-81e3-409a-ae24-163f0646b018-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.104984 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.104967    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545d9172-81e3-409a-ae24-163f0646b018-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.106728 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.106706    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545d9172-81e3-409a-ae24-163f0646b018-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.106915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.106883    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545d9172-81e3-409a-ae24-163f0646b018-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.128093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.128036    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd5mb\" (UniqueName: \"kubernetes.io/projected/545d9172-81e3-409a-ae24-163f0646b018-kube-api-access-nd5mb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2srl5\" (UID: \"545d9172-81e3-409a-ae24-163f0646b018\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.179003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.178976    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:33.478264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:33.478235    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"]
Nov 02 07:33:33.478552 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:33:33.478527    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545d9172_81e3_409a_ae24_163f0646b018.slice/crio-9a135123b93f9195de094dab7de1835a30545bdf28e3fdbdf6a3d835396d9fb7 WatchSource:0}: Error finding container 9a135123b93f9195de094dab7de1835a30545bdf28e3fdbdf6a3d835396d9fb7: Status 404 returned error can't find the container with id 9a135123b93f9195de094dab7de1835a30545bdf28e3fdbdf6a3d835396d9fb7
Nov 02 07:33:34.307766 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:34.307724    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5" event={"ID":"545d9172-81e3-409a-ae24-163f0646b018","Type":"ContainerStarted","Data":"9a135123b93f9195de094dab7de1835a30545bdf28e3fdbdf6a3d835396d9fb7"}
Nov 02 07:33:35.314175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:35.314141    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5" event={"ID":"545d9172-81e3-409a-ae24-163f0646b018","Type":"ContainerStarted","Data":"836ab669b8c54f9b0cee31920a8949d1692c6ad94b2f1314658f83a280cffce1"}
Nov 02 07:33:35.356178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:35.356126    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5" podStartSLOduration=2.08867482 podStartE2EDuration="3.356113287s" totalImagesPullingTime="1.267438467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:33:35.355856609 +0000 UTC m=+566.046183573" watchObservedRunningTime="2026-11-02 07:33:35.356113287 +0000 UTC m=+566.046440251"
Nov 02 07:33:36.179144 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.179114    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:36.387012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.386976    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"]
Nov 02 07:33:36.391273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.391251    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.406168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.406142    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 07:33:36.420369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.420345    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"]
Nov 02 07:33:36.540242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.540177    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9720edfa-6c5a-4522-af50-30700c3310ce-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.540361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.540243    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9720edfa-6c5a-4522-af50-30700c3310ce-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.540361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.540304    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9720edfa-6c5a-4522-af50-30700c3310ce-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.540361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.540338    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvb5k\" (UniqueName: \"kubernetes.io/projected/9720edfa-6c5a-4522-af50-30700c3310ce-kube-api-access-gvb5k\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.540463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.540371    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9720edfa-6c5a-4522-af50-30700c3310ce-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.540463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.540444    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9720edfa-6c5a-4522-af50-30700c3310ce-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.641736 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.641701    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9720edfa-6c5a-4522-af50-30700c3310ce-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.641736 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.641742    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9720edfa-6c5a-4522-af50-30700c3310ce-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.641994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.641772    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvb5k\" (UniqueName: \"kubernetes.io/projected/9720edfa-6c5a-4522-af50-30700c3310ce-kube-api-access-gvb5k\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.641994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.641802    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9720edfa-6c5a-4522-af50-30700c3310ce-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.641994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.641821    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9720edfa-6c5a-4522-af50-30700c3310ce-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.641994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.641844    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9720edfa-6c5a-4522-af50-30700c3310ce-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.642310 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.642279    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9720edfa-6c5a-4522-af50-30700c3310ce-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.642380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.642304    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9720edfa-6c5a-4522-af50-30700c3310ce-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.642380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.642333    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9720edfa-6c5a-4522-af50-30700c3310ce-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.644145 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.644124    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9720edfa-6c5a-4522-af50-30700c3310ce-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.644419 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.644398    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9720edfa-6c5a-4522-af50-30700c3310ce-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.665069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.665042    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvb5k\" (UniqueName: \"kubernetes.io/projected/9720edfa-6c5a-4522-af50-30700c3310ce-kube-api-access-gvb5k\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-96rr4\" (UID: \"9720edfa-6c5a-4522-af50-30700c3310ce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:36.704812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:36.704786    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:37.007903 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:37.007860    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"]
Nov 02 07:33:37.008023 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:33:37.007953    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9720edfa_6c5a_4522_af50_30700c3310ce.slice/crio-c398811baab3eeb027ac1048578bfef667753ccea356bf0f6bd0eac2d5b776f4 WatchSource:0}: Error finding container c398811baab3eeb027ac1048578bfef667753ccea356bf0f6bd0eac2d5b776f4: Status 404 returned error can't find the container with id c398811baab3eeb027ac1048578bfef667753ccea356bf0f6bd0eac2d5b776f4
Nov 02 07:33:37.327313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:37.327264    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4" event={"ID":"9720edfa-6c5a-4522-af50-30700c3310ce","Type":"ContainerStarted","Data":"95455441ad481cc456ee17a5311ef2f75c10f6f5d5a7caaeca3c318553441300"}
Nov 02 07:33:37.327471 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:37.327342    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4" event={"ID":"9720edfa-6c5a-4522-af50-30700c3310ce","Type":"ContainerStarted","Data":"c398811baab3eeb027ac1048578bfef667753ccea356bf0f6bd0eac2d5b776f4"}
Nov 02 07:33:37.374250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:37.374194    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4" podStartSLOduration=1.150723325 podStartE2EDuration="1.374180344s" totalImagesPullingTime="223.457019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:33:37.373297032 +0000 UTC m=+568.063623996" watchObservedRunningTime="2026-11-02 07:33:37.374180344 +0000 UTC m=+568.064507308"
Nov 02 07:33:37.705463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:37.705424    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:43.180239 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:43.180192    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:43.199735 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:43.199701    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:43.392272 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:43.392240    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2srl5"
Nov 02 07:33:44.989021 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:33:44.988968    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:33:45.326988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.326908    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"]
Nov 02 07:33:45.331028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.331006    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.346052 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.346021    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 07:33:45.361615 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.361582    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"]
Nov 02 07:33:45.436947 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.436916    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e002ad6-98cb-4c95-b514-ef87cf9fb134-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.437146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.436967    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rdrh\" (UniqueName: \"kubernetes.io/projected/0e002ad6-98cb-4c95-b514-ef87cf9fb134-kube-api-access-6rdrh\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.437146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.436992    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e002ad6-98cb-4c95-b514-ef87cf9fb134-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.437146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.437105    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e002ad6-98cb-4c95-b514-ef87cf9fb134-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.437304 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.437188    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e002ad6-98cb-4c95-b514-ef87cf9fb134-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.437304 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.437225    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e002ad6-98cb-4c95-b514-ef87cf9fb134-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.538257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.538227    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rdrh\" (UniqueName: \"kubernetes.io/projected/0e002ad6-98cb-4c95-b514-ef87cf9fb134-kube-api-access-6rdrh\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.538435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.538265    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e002ad6-98cb-4c95-b514-ef87cf9fb134-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.538435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.538319    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e002ad6-98cb-4c95-b514-ef87cf9fb134-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.538435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.538385    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e002ad6-98cb-4c95-b514-ef87cf9fb134-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.538435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.538425    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e002ad6-98cb-4c95-b514-ef87cf9fb134-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.538662 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.538453    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e002ad6-98cb-4c95-b514-ef87cf9fb134-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.538851 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.538812    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e002ad6-98cb-4c95-b514-ef87cf9fb134-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.538851 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.538831    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e002ad6-98cb-4c95-b514-ef87cf9fb134-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.539036 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.538875    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e002ad6-98cb-4c95-b514-ef87cf9fb134-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.540748 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.540721    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e002ad6-98cb-4c95-b514-ef87cf9fb134-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.541053 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.541031    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e002ad6-98cb-4c95-b514-ef87cf9fb134-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.561433 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.561410    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rdrh\" (UniqueName: \"kubernetes.io/projected/0e002ad6-98cb-4c95-b514-ef87cf9fb134-kube-api-access-6rdrh\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv\" (UID: \"0e002ad6-98cb-4c95-b514-ef87cf9fb134\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.646229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.646128    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:45.964535 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:33:45.964490    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e002ad6_98cb_4c95_b514_ef87cf9fb134.slice/crio-127fcafcae163f653e77e5b0c1dc1f15e4e7e8338b859a014c82fab99409b3ae WatchSource:0}: Error finding container 127fcafcae163f653e77e5b0c1dc1f15e4e7e8338b859a014c82fab99409b3ae: Status 404 returned error can't find the container with id 127fcafcae163f653e77e5b0c1dc1f15e4e7e8338b859a014c82fab99409b3ae
Nov 02 07:33:45.965713 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:45.965681    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"]
Nov 02 07:33:46.370549 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.370512    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"]
Nov 02 07:33:46.376935 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.376224    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.395493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.395465    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"]
Nov 02 07:33:46.411951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.411230    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 02 07:33:46.427170 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.427142    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv" event={"ID":"0e002ad6-98cb-4c95-b514-ef87cf9fb134","Type":"ContainerStarted","Data":"ced69396fa43ed99faa6bd2632a719ead01ecea569ddf2014c886f60edce0262"}
Nov 02 07:33:46.427292 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.427198    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv" event={"ID":"0e002ad6-98cb-4c95-b514-ef87cf9fb134","Type":"ContainerStarted","Data":"127fcafcae163f653e77e5b0c1dc1f15e4e7e8338b859a014c82fab99409b3ae"}
Nov 02 07:33:46.480342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.480234    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv" podStartSLOduration=1.250204965 podStartE2EDuration="1.480220765s" totalImagesPullingTime="230.0158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:33:46.47913103 +0000 UTC m=+577.169458012" watchObservedRunningTime="2026-11-02 07:33:46.480220765 +0000 UTC m=+577.170547730"
Nov 02 07:33:46.548019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.547954    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147b04eb-2c14-4316-914e-788b949ccb50-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.548608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.548542    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147b04eb-2c14-4316-914e-788b949ccb50-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.548608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.548587    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d97d\" (UniqueName: \"kubernetes.io/projected/147b04eb-2c14-4316-914e-788b949ccb50-kube-api-access-5d97d\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.548792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.548636    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147b04eb-2c14-4316-914e-788b949ccb50-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.548792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.548668    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147b04eb-2c14-4316-914e-788b949ccb50-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.548922 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.548812    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147b04eb-2c14-4316-914e-788b949ccb50-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.646992 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.646948    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:46.649401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.649380    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147b04eb-2c14-4316-914e-788b949ccb50-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.649457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.649410    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147b04eb-2c14-4316-914e-788b949ccb50-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.649509 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.649496    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147b04eb-2c14-4316-914e-788b949ccb50-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.649546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.649528    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d97d\" (UniqueName: \"kubernetes.io/projected/147b04eb-2c14-4316-914e-788b949ccb50-kube-api-access-5d97d\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.649593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.649559    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147b04eb-2c14-4316-914e-788b949ccb50-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.649593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.649587    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147b04eb-2c14-4316-914e-788b949ccb50-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.649967 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.649940    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/147b04eb-2c14-4316-914e-788b949ccb50-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.650114 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.650005    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/147b04eb-2c14-4316-914e-788b949ccb50-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.650114 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.650013    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/147b04eb-2c14-4316-914e-788b949ccb50-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.651699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.651673    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/147b04eb-2c14-4316-914e-788b949ccb50-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.652082 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.652064    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/147b04eb-2c14-4316-914e-788b949ccb50-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.679065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.679033    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d97d\" (UniqueName: \"kubernetes.io/projected/147b04eb-2c14-4316-914e-788b949ccb50-kube-api-access-5d97d\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl\" (UID: \"147b04eb-2c14-4316-914e-788b949ccb50\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:46.707630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.706316    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:46.724215 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.724184    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:46.727358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:46.727326    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:47.052539 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:33:47.052507    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod147b04eb_2c14_4316_914e_788b949ccb50.slice/crio-b23948af4cafea35dd6906a588b29bfc12dd29d1231a082dc286fff82914bc9f WatchSource:0}: Error finding container b23948af4cafea35dd6906a588b29bfc12dd29d1231a082dc286fff82914bc9f: Status 404 returned error can't find the container with id b23948af4cafea35dd6906a588b29bfc12dd29d1231a082dc286fff82914bc9f
Nov 02 07:33:47.053198 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:47.053176    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"]
Nov 02 07:33:47.434365 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:47.434328    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" event={"ID":"147b04eb-2c14-4316-914e-788b949ccb50","Type":"ContainerStarted","Data":"8f38c67862b5f110c97c6d42af0620c2c5f4e4fe13ea967e16028b15e0c6c0c0"}
Nov 02 07:33:47.434365 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:47.434378    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" event={"ID":"147b04eb-2c14-4316-914e-788b949ccb50","Type":"ContainerStarted","Data":"b23948af4cafea35dd6906a588b29bfc12dd29d1231a082dc286fff82914bc9f"}
Nov 02 07:33:47.445864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:47.445834    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-96rr4"
Nov 02 07:33:47.477771 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:47.477716    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podStartSLOduration=1.272695022 podStartE2EDuration="1.477699647s" totalImagesPullingTime="205.004625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:33:47.47616148 +0000 UTC m=+578.166488443" watchObservedRunningTime="2026-11-02 07:33:47.477699647 +0000 UTC m=+578.168026612"
Nov 02 07:33:47.728016 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:47.727966    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:55.647175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:55.647092    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:55.659547 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:55.659521    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:56.498448 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:56.498414    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv"
Nov 02 07:33:56.728003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:56.727964    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:56.739959 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:56.739940    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:33:57.508291 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:33:57.508244    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:34:02.525650 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.525612    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"]
Nov 02 07:34:02.532929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.532875    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.557354 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.557321    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"]
Nov 02 07:34:02.594438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.594399    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.594438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.594438    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64107366-d1ff-46e2-8883-f21ae646ef25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.594672 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.594560    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.594672 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.594612    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54rj7\" (UniqueName: \"kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-kube-api-access-54rj7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.594745 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.594712    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.594745 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.594738    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64107366-d1ff-46e2-8883-f21ae646ef25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.594826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.594766    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.594872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.594856    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.594951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.594917    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.695985    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54rj7\" (UniqueName: \"kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-kube-api-access-54rj7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.696039    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.696057    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64107366-d1ff-46e2-8883-f21ae646ef25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.696083    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.696114    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.696140    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.696183    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.696212    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64107366-d1ff-46e2-8883-f21ae646ef25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.696282    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.696503    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.696530    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.696586    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.696772 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.696696    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.697066 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.697045    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64107366-d1ff-46e2-8883-f21ae646ef25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.698574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.698520    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.698751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.698732    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64107366-d1ff-46e2-8883-f21ae646ef25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.719458 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.719423    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.719592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.719498    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54rj7\" (UniqueName: \"kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-kube-api-access-54rj7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k9x79\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:02.848387 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:02.848297    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:03.162751 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:34:03.162716    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64107366_d1ff_46e2_8883_f21ae646ef25.slice/crio-7773ffac4b870af8e5803036a1ef1d541cf4804213305edf95298cde6599c984 WatchSource:0}: Error finding container 7773ffac4b870af8e5803036a1ef1d541cf4804213305edf95298cde6599c984: Status 404 returned error can't find the container with id 7773ffac4b870af8e5803036a1ef1d541cf4804213305edf95298cde6599c984
Nov 02 07:34:03.163562 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:03.163535    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"]
Nov 02 07:34:03.165405 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:03.165376    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:03.165477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:03.165442    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:03.165521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:03.165474    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:03.529149 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:03.529061    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79" event={"ID":"64107366-d1ff-46e2-8883-f21ae646ef25","Type":"ContainerStarted","Data":"4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3"}
Nov 02 07:34:03.529149 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:03.529118    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79" event={"ID":"64107366-d1ff-46e2-8883-f21ae646ef25","Type":"ContainerStarted","Data":"7773ffac4b870af8e5803036a1ef1d541cf4804213305edf95298cde6599c984"}
Nov 02 07:34:03.592197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:03.592131    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79" podStartSLOduration=1.59211518 podStartE2EDuration="1.59211518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:03.591468751 +0000 UTC m=+594.281795708" watchObservedRunningTime="2026-11-02 07:34:03.59211518 +0000 UTC m=+594.282442143"
Nov 02 07:34:03.849302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:03.849216    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:04.228133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.228096    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5698f55746-5klb6"]
Nov 02 07:34:04.232220 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.232200    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5698f55746-5klb6"
Nov 02 07:34:04.253007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.252976    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5698f55746-5klb6"]
Nov 02 07:34:04.312398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.312352    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf-tls-cert\") pod \"authorino-5698f55746-5klb6\" (UID: \"37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf\") " pod="kuadrant-system/authorino-5698f55746-5klb6"
Nov 02 07:34:04.312398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.312402    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qm6\" (UniqueName: \"kubernetes.io/projected/37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf-kube-api-access-d5qm6\") pod \"authorino-5698f55746-5klb6\" (UID: \"37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf\") " pod="kuadrant-system/authorino-5698f55746-5klb6"
Nov 02 07:34:04.413559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.413505    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf-tls-cert\") pod \"authorino-5698f55746-5klb6\" (UID: \"37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf\") " pod="kuadrant-system/authorino-5698f55746-5klb6"
Nov 02 07:34:04.413559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.413565    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qm6\" (UniqueName: \"kubernetes.io/projected/37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf-kube-api-access-d5qm6\") pod \"authorino-5698f55746-5klb6\" (UID: \"37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf\") " pod="kuadrant-system/authorino-5698f55746-5klb6"
Nov 02 07:34:04.416090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.416059    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf-tls-cert\") pod \"authorino-5698f55746-5klb6\" (UID: \"37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf\") " pod="kuadrant-system/authorino-5698f55746-5klb6"
Nov 02 07:34:04.437563 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.437531    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qm6\" (UniqueName: \"kubernetes.io/projected/37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf-kube-api-access-d5qm6\") pod \"authorino-5698f55746-5klb6\" (UID: \"37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf\") " pod="kuadrant-system/authorino-5698f55746-5klb6"
Nov 02 07:34:04.545380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.545298    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5698f55746-5klb6"
Nov 02 07:34:04.851004 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.850926    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k9x79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 07:34:04.851004 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.850991    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79" podUID="64107366-d1ff-46e2-8883-f21ae646ef25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 07:34:04.882665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:04.882616    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5698f55746-5klb6"]
Nov 02 07:34:04.883767 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:34:04.883739    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37f5a6f2_03e6_4af2_b4a2_4af35c3e5cbf.slice/crio-f78ed4e4fa231f0ef21f09306b7a43ae9c99d5f9715d721559cc20985444a01c WatchSource:0}: Error finding container f78ed4e4fa231f0ef21f09306b7a43ae9c99d5f9715d721559cc20985444a01c: Status 404 returned error can't find the container with id f78ed4e4fa231f0ef21f09306b7a43ae9c99d5f9715d721559cc20985444a01c
Nov 02 07:34:05.541696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:05.541660    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5698f55746-5klb6" event={"ID":"37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf","Type":"ContainerStarted","Data":"f78ed4e4fa231f0ef21f09306b7a43ae9c99d5f9715d721559cc20985444a01c"}
Nov 02 07:34:05.849168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:05.849144    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k9x79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 07:34:05.849551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:05.849196    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79" podUID="64107366-d1ff-46e2-8883-f21ae646ef25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 07:34:06.552910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:06.552027    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5698f55746-5klb6" event={"ID":"37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf","Type":"ContainerStarted","Data":"6ea7908ce1c8f71be4103eb5ba056324033b5c24bd15657497c0d990bebd4700"}
Nov 02 07:34:06.600172 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:06.600086    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5698f55746-5klb6" podStartSLOduration=1.662566772 podStartE2EDuration="2.600066607s" totalImagesPullingTime="937.499835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:06.597048253 +0000 UTC m=+597.287375218" watchObservedRunningTime="2026-11-02 07:34:06.600066607 +0000 UTC m=+597.290393568"
Nov 02 07:34:06.637950 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:06.637746    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7564847bc7-zr86m"]
Nov 02 07:34:06.637950 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:06.637933    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7564847bc7-zr86m" podUID="284c9e7f-d9e9-4ae7-8d8e-4390045d2f51" containerName="authorino" containerID="cri-o://50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51" gracePeriod=30
Nov 02 07:34:06.849491 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:06.849454    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k9x79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 07:34:06.849964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:06.849520    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79" podUID="64107366-d1ff-46e2-8883-f21ae646ef25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 07:34:06.951359 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:06.951322    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7564847bc7-zr86m"
Nov 02 07:34:07.040790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.040748    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-tls-cert\" (UniqueName: \"kubernetes.io/secret/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-tls-cert\") pod \"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51\" (UID: \"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51\") "
Nov 02 07:34:07.041325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.040924    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-kube-api-access-lflwk\" (UniqueName: \"kubernetes.io/projected/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-kube-api-access-lflwk\") pod \"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51\" (UID: \"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51\") "
Nov 02 07:34:07.044425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.044390    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-kube-api-access-lflwk" pod "284c9e7f-d9e9-4ae7-8d8e-4390045d2f51" (UID: "284c9e7f-d9e9-4ae7-8d8e-4390045d2f51"). InnerVolumeSpecName "kube-api-access-lflwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:34:07.091473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.091387    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-tls-cert" pod "284c9e7f-d9e9-4ae7-8d8e-4390045d2f51" (UID: "284c9e7f-d9e9-4ae7-8d8e-4390045d2f51"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:34:07.143264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.143228    4898 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-tls-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:34:07.143264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.143260    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lflwk\" (UniqueName: \"kubernetes.io/projected/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51-kube-api-access-lflwk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:34:07.581912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.581558    4898 generic.go:356] "Generic (PLEG): container finished" podID="284c9e7f-d9e9-4ae7-8d8e-4390045d2f51" containerID="50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51" exitCode=0
Nov 02 07:34:07.585912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.582388    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7564847bc7-zr86m"
Nov 02 07:34:07.585912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.582796    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7564847bc7-zr86m" event={"ID":"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51","Type":"ContainerDied","Data":"50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51"}
Nov 02 07:34:07.585912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.582831    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7564847bc7-zr86m" event={"ID":"284c9e7f-d9e9-4ae7-8d8e-4390045d2f51","Type":"ContainerDied","Data":"1f3c2996b5421ff7ffa0da2eb808dfbf71f236768d6c62d10e8bab88b0664ced"}
Nov 02 07:34:07.585912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.582853    4898 scope.go:122] "RemoveContainer" containerID="50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51"
Nov 02 07:34:07.605319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.605285    4898 scope.go:122] "RemoveContainer" containerID="50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51"
Nov 02 07:34:07.606028 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:34:07.605995    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51\": container with ID starting with 50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51 not found: ID does not exist" containerID="50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51"
Nov 02 07:34:07.606160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.606036    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51"} err="failed to get container status \"50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51\": rpc error: code = NotFound desc = could not find container \"50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51\": container with ID starting with 50dbc7c91aaa2a01c2a94b42704aba7999caf6255640d7c6a2343a00e4bd2d51 not found: ID does not exist"
Nov 02 07:34:07.635427 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.635390    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7564847bc7-zr86m"]
Nov 02 07:34:07.647429 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.647395    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7564847bc7-zr86m"]
Nov 02 07:34:07.849148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.849051    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k9x79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 07:34:07.849148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:07.849117    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79" podUID="64107366-d1ff-46e2-8883-f21ae646ef25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 07:34:08.849718 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:08.849678    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k9x79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 07:34:08.850143 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:08.849752    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79" podUID="64107366-d1ff-46e2-8883-f21ae646ef25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 07:34:09.465913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:09.465125    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284c9e7f-d9e9-4ae7-8d8e-4390045d2f51" path="/var/lib/kubelet/pods/284c9e7f-d9e9-4ae7-8d8e-4390045d2f51/volumes"
Nov 02 07:34:09.637848 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:09.637818    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:34:09.648698 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:09.648668    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:34:09.849716 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:09.849624    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k9x79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 07:34:09.849716 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:09.849696    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79" podUID="64107366-d1ff-46e2-8883-f21ae646ef25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 07:34:11.709743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:11.709708    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:11.710255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:11.710226    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:11.711071 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:11.711040    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:34:14.989530 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:34:14.989482    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:34:17.563182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.563148    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"]
Nov 02 07:34:17.563606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.563580    4898 cpu_manager.go:414] "containerMap: removing container" podUID="284c9e7f-d9e9-4ae7-8d8e-4390045d2f51" containerName="authorino"
Nov 02 07:34:17.563606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.563592    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284c9e7f-d9e9-4ae7-8d8e-4390045d2f51" containerName="authorino"
Nov 02 07:34:17.563686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.563669    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="284c9e7f-d9e9-4ae7-8d8e-4390045d2f51" containerName="authorino"
Nov 02 07:34:17.568393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.568373    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.601017 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.600989    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"]
Nov 02 07:34:17.686631 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.686594    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.686631 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.686639    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.686876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.686768    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.686876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.686852    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkxv4\" (UniqueName: \"kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-kube-api-access-jkxv4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.686985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.686885    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.686985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.686975    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.687059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.687028    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.687119 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.687104    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.687158 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.687131    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.787885 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.787848    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.787918    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkxv4\" (UniqueName: \"kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-kube-api-access-jkxv4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.787952    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.787999    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.788076    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.788128    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.788162    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.788226    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.788254    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788451 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.788365    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788510 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.788468    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.788539    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788629 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.788578    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.788870 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.788845    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.790511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.790488    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.790609 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.790556    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.810961 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.810929    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkxv4\" (UniqueName: \"kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-kube-api-access-jkxv4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.811063 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.811014    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-b652g\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:17.883332 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:17.883252    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:18.194962 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:34:18.194930    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4e4ee9_0e4d_478f_9a88_8dc4c082150e.slice/crio-ad4451f590527222446c4dc801df4931ac3c9a8ec79405bc4d5dead8439feccd WatchSource:0}: Error finding container ad4451f590527222446c4dc801df4931ac3c9a8ec79405bc4d5dead8439feccd: Status 404 returned error can't find the container with id ad4451f590527222446c4dc801df4931ac3c9a8ec79405bc4d5dead8439feccd
Nov 02 07:34:18.195517 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:18.195494    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"]
Nov 02 07:34:18.196901 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:18.196858    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:18.196978 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:18.196959    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:18.197020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:18.196992    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:18.654705 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:18.654666    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g" event={"ID":"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e","Type":"ContainerStarted","Data":"073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071"}
Nov 02 07:34:18.655228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:18.654743    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g" event={"ID":"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e","Type":"ContainerStarted","Data":"ad4451f590527222446c4dc801df4931ac3c9a8ec79405bc4d5dead8439feccd"}
Nov 02 07:34:18.711866 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:18.711809    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g" podStartSLOduration=1.711797136 podStartE2EDuration="1.711797136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:18.711206863 +0000 UTC m=+609.401533864" watchObservedRunningTime="2026-11-02 07:34:18.711797136 +0000 UTC m=+609.402124100"
Nov 02 07:34:18.883900 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:18.883847    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:19.885568 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:19.885500    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-b652g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 07:34:19.885568 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:19.885552    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g" podUID="1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 07:34:20.884236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:20.884198    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-b652g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 07:34:20.884417 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:20.884256    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g" podUID="1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 07:34:21.884446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:21.884406    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-b652g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 07:34:21.884870 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:21.884478    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g" podUID="1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 07:34:22.884359 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:22.884315    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-b652g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 07:34:22.884538 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:22.884380    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g" podUID="1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 07:34:23.884492 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:23.884448    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-b652g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 07:34:23.884672 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:23.884509    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g" podUID="1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 07:34:24.884431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:24.884391    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-b652g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 07:34:24.884619 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:24.884452    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g" podUID="1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 07:34:26.379389 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:26.379351    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:26.379872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:26.379725    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:26.380440 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:26.380422    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:34:31.412303 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.412263    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"]
Nov 02 07:34:31.429456 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.429417    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.442022 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.441984    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"]
Nov 02 07:34:31.450990 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.450963    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-be1fb7c9-openshift-default-dockercfg-nfhlx\""
Nov 02 07:34:31.530675 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.530638    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc9wm\" (UniqueName: \"kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-kube-api-access-cc9wm\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.530878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.530702    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-socket\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.530878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.530800    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-credential-socket\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.530878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.530829    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-envoy\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.530878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.530860    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b59f3-7992-4441-a070-3efb6bf36cb6-istiod-ca-cert\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.531129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.531019    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-token\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.531129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.531093    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-podinfo\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.531238 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.531140    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-certs\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.531295 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.531228    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-data\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.632633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.632595    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-token\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.632814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.632659    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-podinfo\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.632814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.632697    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-certs\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.632930 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.632830    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-data\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.633005 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.632950    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc9wm\" (UniqueName: \"kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-kube-api-access-cc9wm\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.633005 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.632980    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-socket\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.633005 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.633002    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-credential-socket\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.633164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.633022    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-envoy\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.633164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.633051    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b59f3-7992-4441-a070-3efb6bf36cb6-istiod-ca-cert\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.633274 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.633169    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-certs\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.633274 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.633225    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-data\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.633468 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.633447    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-credential-socket\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.633531 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.633468    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-socket\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.633712 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.633691    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b59f3-7992-4441-a070-3efb6bf36cb6-istiod-ca-cert\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.635216 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.635192    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-envoy\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.635346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.635312    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-podinfo\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.658844 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.658813    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc9wm\" (UniqueName: \"kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-kube-api-access-cc9wm\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.658844 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.658827    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-token\") pod \"e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") " pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.687310 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.687222    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"]
Nov 02 07:34:31.692296 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.692263    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.716073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.716034    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"]
Nov 02 07:34:31.716257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.716149    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b9780053-gw-openshift-default-dockercfg-8cmqz\""
Nov 02 07:34:31.747515 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.747478    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:31.835754 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.835714    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-data\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.836006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.835780    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e779948-dde1-40c4-bbe7-7a365344f1ed-istiod-ca-cert\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.836077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.836055    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-certs\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.836136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.836114    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-socket\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.836197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.836149    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-token\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.836197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.836187    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf9cj\" (UniqueName: \"kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-kube-api-access-bf9cj\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.836294 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.836234    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-credential-socket\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.836294 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.836258    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-podinfo\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.836389 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.836351    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-envoy\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.937295 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.937259    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-data\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.937295 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.937299    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e779948-dde1-40c4-bbe7-7a365344f1ed-istiod-ca-cert\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.937582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.937432    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-certs\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.937582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.937511    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-socket\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.937582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.937543    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-token\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.937582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.937578    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf9cj\" (UniqueName: \"kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-kube-api-access-bf9cj\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.937813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.937619    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-credential-socket\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.937813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.937641    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-podinfo\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.937813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.937737    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-envoy\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.937813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.937750    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-data\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.938019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.937835    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-certs\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.938071 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.938048    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-socket\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.938107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.938091    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-credential-socket\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.938187 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.938171    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e779948-dde1-40c4-bbe7-7a365344f1ed-istiod-ca-cert\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.940157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.940126    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-envoy\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.940353 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.940335    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-podinfo\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.961051 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.961020    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-token\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:31.961179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:31.961141    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf9cj\" (UniqueName: \"kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-kube-api-access-bf9cj\") pod \"e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") " pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:32.008987 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.008951    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:32.081792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.081759    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"]
Nov 02 07:34:32.082981 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:34:32.082957    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569b59f3_7992_4441_a070_3efb6bf36cb6.slice/crio-1a2a1b4b4f1f66ffc903a8937eb5a412ce69c2e377597bb13be3f5c27aabb11d WatchSource:0}: Error finding container 1a2a1b4b4f1f66ffc903a8937eb5a412ce69c2e377597bb13be3f5c27aabb11d: Status 404 returned error can't find the container with id 1a2a1b4b4f1f66ffc903a8937eb5a412ce69c2e377597bb13be3f5c27aabb11d
Nov 02 07:34:32.085311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.085276    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:32.085412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.085342    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:32.085412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.085370    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:32.315944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.315043    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"]
Nov 02 07:34:32.320186 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.320149    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:32.320309 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.320236    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:32.320309 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.320285    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:32.759242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.759202    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" event={"ID":"6e779948-dde1-40c4-bbe7-7a365344f1ed","Type":"ContainerStarted","Data":"c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845"}
Nov 02 07:34:32.759711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.759262    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" event={"ID":"6e779948-dde1-40c4-bbe7-7a365344f1ed","Type":"ContainerStarted","Data":"ea902552b586a1081cca67a3c4236dec78ad929120183576f5b7269aea4f9c6c"}
Nov 02 07:34:32.760995 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.760961    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" event={"ID":"569b59f3-7992-4441-a070-3efb6bf36cb6","Type":"ContainerStarted","Data":"0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016"}
Nov 02 07:34:32.761129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.761014    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" event={"ID":"569b59f3-7992-4441-a070-3efb6bf36cb6","Type":"ContainerStarted","Data":"1a2a1b4b4f1f66ffc903a8937eb5a412ce69c2e377597bb13be3f5c27aabb11d"}
Nov 02 07:34:32.806876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.806826    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" podStartSLOduration=1.806811013 podStartE2EDuration="1.806811013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:32.804569784 +0000 UTC m=+623.494896748" watchObservedRunningTime="2026-11-02 07:34:32.806811013 +0000 UTC m=+623.497137976"
Nov 02 07:34:32.852068 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:32.852003    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" podStartSLOduration=1.851974016 podStartE2EDuration="1.851974016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:32.850461859 +0000 UTC m=+623.540788845" watchObservedRunningTime="2026-11-02 07:34:32.851974016 +0000 UTC m=+623.542300994"
Nov 02 07:34:33.013495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:33.013394    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:33.024233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:33.024185    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:33.747760 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:33.747707    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:33.753129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:33.753100    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:33.769878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:33.769848    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:33.770319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:33.770117    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:33.770941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:33.770923    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:33.770990 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:33.770969    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:34:37.937512 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:37.937480    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"]
Nov 02 07:34:37.942401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:37.942379    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"
Nov 02 07:34:37.977063 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:37.976991    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"]
Nov 02 07:34:38.000775 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.000741    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e84d20-8952-43a9-a57c-21b2ab0f1221-plugins-config-volume\") pod \"payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k\" (UID: \"b6e84d20-8952-43a9-a57c-21b2ab0f1221\") " pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"
Nov 02 07:34:38.000992 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.000954    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfr5w\" (UniqueName: \"kubernetes.io/projected/b6e84d20-8952-43a9-a57c-21b2ab0f1221-kube-api-access-hfr5w\") pod \"payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k\" (UID: \"b6e84d20-8952-43a9-a57c-21b2ab0f1221\") " pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"
Nov 02 07:34:38.005603 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.005579    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-935d4e21-dockercfg-c9cmh\""
Nov 02 07:34:38.027407 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.027373    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-935d4e21\""
Nov 02 07:34:38.102493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.102453    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfr5w\" (UniqueName: \"kubernetes.io/projected/b6e84d20-8952-43a9-a57c-21b2ab0f1221-kube-api-access-hfr5w\") pod \"payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k\" (UID: \"b6e84d20-8952-43a9-a57c-21b2ab0f1221\") " pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"
Nov 02 07:34:38.102669 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.102517    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e84d20-8952-43a9-a57c-21b2ab0f1221-plugins-config-volume\") pod \"payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k\" (UID: \"b6e84d20-8952-43a9-a57c-21b2ab0f1221\") " pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"
Nov 02 07:34:38.103239 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.103220    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e84d20-8952-43a9-a57c-21b2ab0f1221-plugins-config-volume\") pod \"payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k\" (UID: \"b6e84d20-8952-43a9-a57c-21b2ab0f1221\") " pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"
Nov 02 07:34:38.126802 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.126769    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfr5w\" (UniqueName: \"kubernetes.io/projected/b6e84d20-8952-43a9-a57c-21b2ab0f1221-kube-api-access-hfr5w\") pod \"payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k\" (UID: \"b6e84d20-8952-43a9-a57c-21b2ab0f1221\") " pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"
Nov 02 07:34:38.257518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.257431    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"
Nov 02 07:34:38.554958 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:34:38.554922    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e84d20_8952_43a9_a57c_21b2ab0f1221.slice/crio-69859a68f9c4a12ddfa74ba06eb13516b526b44b23214f609502ac5d9ba66cc4 WatchSource:0}: Error finding container 69859a68f9c4a12ddfa74ba06eb13516b526b44b23214f609502ac5d9ba66cc4: Status 404 returned error can't find the container with id 69859a68f9c4a12ddfa74ba06eb13516b526b44b23214f609502ac5d9ba66cc4
Nov 02 07:34:38.555183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.555159    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"]
Nov 02 07:34:38.556782 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.556766    4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 07:34:38.797436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.797399    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k" event={"ID":"b6e84d20-8952-43a9-a57c-21b2ab0f1221","Type":"ContainerStarted","Data":"69859a68f9c4a12ddfa74ba06eb13516b526b44b23214f609502ac5d9ba66cc4"}
Nov 02 07:34:38.913668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.913617    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"]
Nov 02 07:34:38.918794 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.918766    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"
Nov 02 07:34:38.941337 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.941306    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"]
Nov 02 07:34:38.953441 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.953410    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-be1fb7c9\""
Nov 02 07:34:38.953661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:38.953636    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-be1fb7c9-dockercfg-6s8lv\""
Nov 02 07:34:39.013957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.013877    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qwl6\" (UniqueName: \"kubernetes.io/projected/151f52fe-5ecf-417a-8f40-ebffd15ea385-kube-api-access-2qwl6\") pod \"payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5\" (UID: \"151f52fe-5ecf-417a-8f40-ebffd15ea385\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"
Nov 02 07:34:39.014300 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.014265    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151f52fe-5ecf-417a-8f40-ebffd15ea385-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5\" (UID: \"151f52fe-5ecf-417a-8f40-ebffd15ea385\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"
Nov 02 07:34:39.115071 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.115028    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151f52fe-5ecf-417a-8f40-ebffd15ea385-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5\" (UID: \"151f52fe-5ecf-417a-8f40-ebffd15ea385\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"
Nov 02 07:34:39.115268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.115134    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qwl6\" (UniqueName: \"kubernetes.io/projected/151f52fe-5ecf-417a-8f40-ebffd15ea385-kube-api-access-2qwl6\") pod \"payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5\" (UID: \"151f52fe-5ecf-417a-8f40-ebffd15ea385\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"
Nov 02 07:34:39.115769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.115740    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151f52fe-5ecf-417a-8f40-ebffd15ea385-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5\" (UID: \"151f52fe-5ecf-417a-8f40-ebffd15ea385\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"
Nov 02 07:34:39.140254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.140210    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"]
Nov 02 07:34:39.144969 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.144460    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.146585 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.146561    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qwl6\" (UniqueName: \"kubernetes.io/projected/151f52fe-5ecf-417a-8f40-ebffd15ea385-kube-api-access-2qwl6\") pod \"payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5\" (UID: \"151f52fe-5ecf-417a-8f40-ebffd15ea385\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"
Nov 02 07:34:39.171420 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.169084    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8c43c08e-openshift-default-dockercfg-nfdqp\""
Nov 02 07:34:39.172916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.172877    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"]
Nov 02 07:34:39.217093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.216735    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-envoy\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.217093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.216854    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5jn\" (UniqueName: \"kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-kube-api-access-dk5jn\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.217093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.216963    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-token\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.217093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.217008    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-podinfo\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.217093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.217060    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-credential-socket\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.217093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.217085    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-socket\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.217093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.217107    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5445332-4238-4877-a4d1-073fa1cb3bbd-istiod-ca-cert\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.217567 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.217167    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-data\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.217567 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.217184    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-certs\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.233502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.233473    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.318239    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-data\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.318286    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-certs\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.318358    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-envoy\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.318385    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5jn\" (UniqueName: \"kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-kube-api-access-dk5jn\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.318411    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-token\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.318469    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-podinfo\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.318531    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-credential-socket\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.318569    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-socket\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.318597    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5445332-4238-4877-a4d1-073fa1cb3bbd-istiod-ca-cert\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.319642    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-data\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.319932    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-certs\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.320647    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-credential-socket\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.320915    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-socket\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.321665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.321571    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5445332-4238-4877-a4d1-073fa1cb3bbd-istiod-ca-cert\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.323063 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.323020    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-podinfo\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.343930 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.343875    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5jn\" (UniqueName: \"kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-kube-api-access-dk5jn\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.345287 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.345256    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-token\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.348630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.348576    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-envoy\") pod \"e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") " pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.481553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.481463    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:39.544836 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:34:39.543819    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151f52fe_5ecf_417a_8f40_ebffd15ea385.slice/crio-c4f9f5457d2af3e5d5000c56a854b2272b0a0b872cde97d4c242fcfb6bac5927 WatchSource:0}: Error finding container c4f9f5457d2af3e5d5000c56a854b2272b0a0b872cde97d4c242fcfb6bac5927: Status 404 returned error can't find the container with id c4f9f5457d2af3e5d5000c56a854b2272b0a0b872cde97d4c242fcfb6bac5927
Nov 02 07:34:39.544836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.544043    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"]
Nov 02 07:34:39.806718 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.806639    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5" event={"ID":"151f52fe-5ecf-417a-8f40-ebffd15ea385","Type":"ContainerStarted","Data":"c4f9f5457d2af3e5d5000c56a854b2272b0a0b872cde97d4c242fcfb6bac5927"}
Nov 02 07:34:39.825073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.825042    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"]
Nov 02 07:34:39.825640 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:34:39.825610    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5445332_4238_4877_a4d1_073fa1cb3bbd.slice/crio-522e296175e324ccbb2d602ff15190e768de1af325d5b4318865ff1f88be98ca WatchSource:0}: Error finding container 522e296175e324ccbb2d602ff15190e768de1af325d5b4318865ff1f88be98ca: Status 404 returned error can't find the container with id 522e296175e324ccbb2d602ff15190e768de1af325d5b4318865ff1f88be98ca
Nov 02 07:34:39.829631 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.829597    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:39.829744 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.829702    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:39.829814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:39.829751    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:40.817000 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:40.816942    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" event={"ID":"c5445332-4238-4877-a4d1-073fa1cb3bbd","Type":"ContainerStarted","Data":"cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550"}
Nov 02 07:34:40.817503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:40.817024    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" event={"ID":"c5445332-4238-4877-a4d1-073fa1cb3bbd","Type":"ContainerStarted","Data":"522e296175e324ccbb2d602ff15190e768de1af325d5b4318865ff1f88be98ca"}
Nov 02 07:34:40.886774 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:40.886716    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" podStartSLOduration=1.886696011 podStartE2EDuration="1.886696011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:40.873791099 +0000 UTC m=+631.564118102" watchObservedRunningTime="2026-11-02 07:34:40.886696011 +0000 UTC m=+631.577022975"
Nov 02 07:34:41.307906 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.307867    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"]
Nov 02 07:34:41.312191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.312161    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"
Nov 02 07:34:41.334390 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.334355    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8ff8e809\""
Nov 02 07:34:41.335512 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.334758    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8ff8e809-dockercfg-nhd8t\""
Nov 02 07:34:41.340903 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.335678    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"]
Nov 02 07:34:41.347069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.347041    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npnfx\" (UniqueName: \"kubernetes.io/projected/507a969f-d4b9-461e-a388-e8ff376c766e-kube-api-access-npnfx\") pod \"payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm\" (UID: \"507a969f-d4b9-461e-a388-e8ff376c766e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"
Nov 02 07:34:41.347204 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.347109    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507a969f-d4b9-461e-a388-e8ff376c766e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm\" (UID: \"507a969f-d4b9-461e-a388-e8ff376c766e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"
Nov 02 07:34:41.450109 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.448382    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npnfx\" (UniqueName: \"kubernetes.io/projected/507a969f-d4b9-461e-a388-e8ff376c766e-kube-api-access-npnfx\") pod \"payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm\" (UID: \"507a969f-d4b9-461e-a388-e8ff376c766e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"
Nov 02 07:34:41.450109 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.448491    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507a969f-d4b9-461e-a388-e8ff376c766e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm\" (UID: \"507a969f-d4b9-461e-a388-e8ff376c766e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"
Nov 02 07:34:41.450109 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.449913    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507a969f-d4b9-461e-a388-e8ff376c766e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm\" (UID: \"507a969f-d4b9-461e-a388-e8ff376c766e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"
Nov 02 07:34:41.477853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.477767    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npnfx\" (UniqueName: \"kubernetes.io/projected/507a969f-d4b9-461e-a388-e8ff376c766e-kube-api-access-npnfx\") pod \"payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm\" (UID: \"507a969f-d4b9-461e-a388-e8ff376c766e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"
Nov 02 07:34:41.482438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.482410    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:41.491419 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.491388    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:41.630056 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.630004    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"
Nov 02 07:34:41.827539 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.827441    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k" event={"ID":"b6e84d20-8952-43a9-a57c-21b2ab0f1221","Type":"ContainerStarted","Data":"a5811f053fb3f52443ddcd7f793582377fa2a6264d8c2e6d80ab18db50a8e148"}
Nov 02 07:34:41.828047 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.827919    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"
Nov 02 07:34:41.828047 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.827959    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:41.835931 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.829044    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:34:41.869517 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.869467    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k" podStartSLOduration=2.200487596 podStartE2EDuration="4.869453268s" totalImagesPullingTime="2.668965672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:41.868127166 +0000 UTC m=+632.558454131" watchObservedRunningTime="2026-11-02 07:34:41.869453268 +0000 UTC m=+632.559780233"
Nov 02 07:34:41.942015 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:34:41.941981    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507a969f_d4b9_461e_a388_e8ff376c766e.slice/crio-41b88368c3b3467dc60a4392148901890ac0d3cfe325e1d9563096126d78e7e4 WatchSource:0}: Error finding container 41b88368c3b3467dc60a4392148901890ac0d3cfe325e1d9563096126d78e7e4: Status 404 returned error can't find the container with id 41b88368c3b3467dc60a4392148901890ac0d3cfe325e1d9563096126d78e7e4
Nov 02 07:34:41.942884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:41.942856    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"]
Nov 02 07:34:42.872392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:42.872347    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5" event={"ID":"151f52fe-5ecf-417a-8f40-ebffd15ea385","Type":"ContainerStarted","Data":"20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc"}
Nov 02 07:34:42.873969 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:42.873943    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"
Nov 02 07:34:42.893699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:42.893657    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm" event={"ID":"507a969f-d4b9-461e-a388-e8ff376c766e","Type":"ContainerStarted","Data":"f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6"}
Nov 02 07:34:42.894011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:42.893987    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm" event={"ID":"507a969f-d4b9-461e-a388-e8ff376c766e","Type":"ContainerStarted","Data":"41b88368c3b3467dc60a4392148901890ac0d3cfe325e1d9563096126d78e7e4"}
Nov 02 07:34:42.894149 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:42.894128    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"
Nov 02 07:34:42.975914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:42.969822    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm" podStartSLOduration=1.9698013410000001 podStartE2EDuration="1.969801341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:42.967336112 +0000 UTC m=+633.657663096" watchObservedRunningTime="2026-11-02 07:34:42.969801341 +0000 UTC m=+633.660128307"
Nov 02 07:34:42.975914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:42.971198    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5" podStartSLOduration=2.687769442 podStartE2EDuration="4.971184588s" totalImagesPullingTime="2.283415146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:42.922107171 +0000 UTC m=+633.612434134" watchObservedRunningTime="2026-11-02 07:34:42.971184588 +0000 UTC m=+633.661511554"
Nov 02 07:34:45.011943 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:34:45.011877    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:34:45.328104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:45.328022    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"]
Nov 02 07:34:45.332393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:45.332366    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"
Nov 02 07:34:45.377915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:45.367164    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8c43c08e-dockercfg-4jscs\""
Nov 02 07:34:45.377915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:45.367283    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8c43c08e\""
Nov 02 07:34:45.385760 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:45.385721    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"]
Nov 02 07:34:45.446824 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:45.446784    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5hnr\" (UniqueName: \"kubernetes.io/projected/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-kube-api-access-l5hnr\") pod \"payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx\" (UID: \"fd3e3dc4-09fe-497e-92a2-21297c58aa1c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"
Nov 02 07:34:45.447036 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:45.446867    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx\" (UID: \"fd3e3dc4-09fe-497e-92a2-21297c58aa1c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"
Nov 02 07:34:45.548217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:45.548177    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx\" (UID: \"fd3e3dc4-09fe-497e-92a2-21297c58aa1c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"
Nov 02 07:34:45.548395 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:45.548302    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5hnr\" (UniqueName: \"kubernetes.io/projected/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-kube-api-access-l5hnr\") pod \"payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx\" (UID: \"fd3e3dc4-09fe-497e-92a2-21297c58aa1c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"
Nov 02 07:34:45.548999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:45.548979    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx\" (UID: \"fd3e3dc4-09fe-497e-92a2-21297c58aa1c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"
Nov 02 07:34:45.582113 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:45.582010    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5hnr\" (UniqueName: \"kubernetes.io/projected/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-kube-api-access-l5hnr\") pod \"payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx\" (UID: \"fd3e3dc4-09fe-497e-92a2-21297c58aa1c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"
Nov 02 07:34:45.647019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:45.646984    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"
Nov 02 07:34:46.033412 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:34:46.033380    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3e3dc4_09fe_497e_92a2_21297c58aa1c.slice/crio-807e63965db3ba975dd2d88364c4e1638bc757a4211b6de8908882a612e73fd5 WatchSource:0}: Error finding container 807e63965db3ba975dd2d88364c4e1638bc757a4211b6de8908882a612e73fd5: Status 404 returned error can't find the container with id 807e63965db3ba975dd2d88364c4e1638bc757a4211b6de8908882a612e73fd5
Nov 02 07:34:46.033827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:46.033791    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"]
Nov 02 07:34:46.928677 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:46.928638    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx" event={"ID":"fd3e3dc4-09fe-497e-92a2-21297c58aa1c","Type":"ContainerStarted","Data":"ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390"}
Nov 02 07:34:46.928857 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:46.928700    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx" event={"ID":"fd3e3dc4-09fe-497e-92a2-21297c58aa1c","Type":"ContainerStarted","Data":"807e63965db3ba975dd2d88364c4e1638bc757a4211b6de8908882a612e73fd5"}
Nov 02 07:34:46.928941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:46.928914    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"
Nov 02 07:34:49.816074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:49.816029    4898 patch_prober.go:29] interesting pod/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:34:49.816697 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:49.816108    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:34:51.959868 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:51.959836    4898 generic.go:356] "Generic (PLEG): container finished" podID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerID="c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845" exitCode=0
Nov 02 07:34:51.960361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:51.959914    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" event={"ID":"6e779948-dde1-40c4-bbe7-7a365344f1ed","Type":"ContainerDied","Data":"c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845"}
Nov 02 07:34:51.960361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:51.960250    4898 scope.go:122] "RemoveContainer" containerID="c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845"
Nov 02 07:34:51.961469 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:51.961438    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:51.961535 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:51.961510    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:51.961573 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:51.961555    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:34:52.020726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:52.020679    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx" podStartSLOduration=7.020666031 podStartE2EDuration="7.020666031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:34:46.988579288 +0000 UTC m=+637.678906264" watchObservedRunningTime="2026-11-02 07:34:52.020666031 +0000 UTC m=+642.710993006"
Nov 02 07:34:52.897234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:52.897203    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"
Nov 02 07:34:52.999522 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:52.999481    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" event={"ID":"6e779948-dde1-40c4-bbe7-7a365344f1ed","Type":"ContainerStarted","Data":"1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763"}
Nov 02 07:34:53.010157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:53.010115    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:53.010314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:53.010283    4898 patch_prober.go:29] interesting pod/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 02 07:34:53.010357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:53.010326    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 02 07:34:53.911513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:53.911482    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"
Nov 02 07:34:54.009589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.009531    4898 patch_prober.go:29] interesting pod/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 02 07:34:54.010005 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.009618    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 02 07:34:54.708641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.708596    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"]
Nov 02 07:34:54.712949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.712924    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:34:54.722357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.722326    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"]
Nov 02 07:34:54.732603 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.731752    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:34:54.732603 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.731782    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:34:54.732603 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.731837    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:34:54.739666 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.739647    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:34:54.817600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.817570    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a892982-23b0-4909-b72b-4822102991ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8\" (UID: \"8a892982-23b0-4909-b72b-4822102991ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:34:54.817828 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.817812    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jchn5\" (UniqueName: \"kubernetes.io/projected/8a892982-23b0-4909-b72b-4822102991ea-kube-api-access-jchn5\") pod \"maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8\" (UID: \"8a892982-23b0-4909-b72b-4822102991ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:34:54.817955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.817939    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a892982-23b0-4909-b72b-4822102991ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8\" (UID: \"8a892982-23b0-4909-b72b-4822102991ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:34:54.916323 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.916295    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"
Nov 02 07:34:54.919206 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.919178    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a892982-23b0-4909-b72b-4822102991ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8\" (UID: \"8a892982-23b0-4909-b72b-4822102991ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:34:54.919330 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.919273    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a892982-23b0-4909-b72b-4822102991ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8\" (UID: \"8a892982-23b0-4909-b72b-4822102991ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:34:54.919330 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.919320    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jchn5\" (UniqueName: \"kubernetes.io/projected/8a892982-23b0-4909-b72b-4822102991ea-kube-api-access-jchn5\") pod \"maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8\" (UID: \"8a892982-23b0-4909-b72b-4822102991ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:34:54.920095 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.920066    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a892982-23b0-4909-b72b-4822102991ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8\" (UID: \"8a892982-23b0-4909-b72b-4822102991ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:34:54.921731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.921706    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a892982-23b0-4909-b72b-4822102991ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8\" (UID: \"8a892982-23b0-4909-b72b-4822102991ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:34:54.942966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:54.942932    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jchn5\" (UniqueName: \"kubernetes.io/projected/8a892982-23b0-4909-b72b-4822102991ea-kube-api-access-jchn5\") pod \"maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8\" (UID: \"8a892982-23b0-4909-b72b-4822102991ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:34:55.015922 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:55.015813    4898 patch_prober.go:29] interesting pod/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 02 07:34:55.016521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:55.016484    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 02 07:34:55.027008 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:55.026920    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:34:55.421965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:55.417043    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"]
Nov 02 07:34:55.423372 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:34:55.423341    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a892982_23b0_4909_b72b_4822102991ea.slice/crio-7fb17c921ca86dc0abf5f5d11b6b2fb403bdb2f64c368d385815b0feef01a1fc WatchSource:0}: Error finding container 7fb17c921ca86dc0abf5f5d11b6b2fb403bdb2f64c368d385815b0feef01a1fc: Status 404 returned error can't find the container with id 7fb17c921ca86dc0abf5f5d11b6b2fb403bdb2f64c368d385815b0feef01a1fc
Nov 02 07:34:56.009569 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.009529    4898 patch_prober.go:29] interesting pod/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 02 07:34:56.009747 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.009600    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 02 07:34:56.030167 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.030137    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8" event={"ID":"8a892982-23b0-4909-b72b-4822102991ea","Type":"ContainerStarted","Data":"7fb17c921ca86dc0abf5f5d11b6b2fb403bdb2f64c368d385815b0feef01a1fc"}
Nov 02 07:34:56.437387 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.437350    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"]
Nov 02 07:34:56.444096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.444076    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:34:56.524554 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.519339    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"]
Nov 02 07:34:56.536150 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.536109    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvlvk\" (UniqueName: \"kubernetes.io/projected/7a4c197d-accd-4564-a8fc-a2b2e1958846-kube-api-access-fvlvk\") pod \"maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2\" (UID: \"7a4c197d-accd-4564-a8fc-a2b2e1958846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:34:56.536317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.536214    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2\" (UID: \"7a4c197d-accd-4564-a8fc-a2b2e1958846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:34:56.536317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.536289    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2\" (UID: \"7a4c197d-accd-4564-a8fc-a2b2e1958846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:34:56.637024 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.636989    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2\" (UID: \"7a4c197d-accd-4564-a8fc-a2b2e1958846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:34:56.637183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.637040    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2\" (UID: \"7a4c197d-accd-4564-a8fc-a2b2e1958846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:34:56.637183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.637109    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvlvk\" (UniqueName: \"kubernetes.io/projected/7a4c197d-accd-4564-a8fc-a2b2e1958846-kube-api-access-fvlvk\") pod \"maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2\" (UID: \"7a4c197d-accd-4564-a8fc-a2b2e1958846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:34:56.638380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.638343    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2\" (UID: \"7a4c197d-accd-4564-a8fc-a2b2e1958846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:34:56.644361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.644288    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2\" (UID: \"7a4c197d-accd-4564-a8fc-a2b2e1958846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:34:56.694955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.685588    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvlvk\" (UniqueName: \"kubernetes.io/projected/7a4c197d-accd-4564-a8fc-a2b2e1958846-kube-api-access-fvlvk\") pod \"maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2\" (UID: \"7a4c197d-accd-4564-a8fc-a2b2e1958846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:34:56.787106 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:56.787066    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:34:57.016654 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:57.016151    4898 patch_prober.go:29] interesting pod/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 02 07:34:57.016654 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:57.016213    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 02 07:34:57.222497 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:57.222460    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"]
Nov 02 07:34:57.225293 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:34:57.225231    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a4c197d_accd_4564_a8fc_a2b2e1958846.slice/crio-3deb8d9637db3c8908b8a96f5aaf92195252ebdb5b7c08c5d28a635c42febd76 WatchSource:0}: Error finding container 3deb8d9637db3c8908b8a96f5aaf92195252ebdb5b7c08c5d28a635c42febd76: Status 404 returned error can't find the container with id 3deb8d9637db3c8908b8a96f5aaf92195252ebdb5b7c08c5d28a635c42febd76
Nov 02 07:34:57.941537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:57.941510    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"
Nov 02 07:34:58.018477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:58.018004    4898 patch_prober.go:29] interesting pod/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 02 07:34:58.018477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:58.018069    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 02 07:34:58.052901 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:58.052815    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2" event={"ID":"7a4c197d-accd-4564-a8fc-a2b2e1958846","Type":"ContainerStarted","Data":"3deb8d9637db3c8908b8a96f5aaf92195252ebdb5b7c08c5d28a635c42febd76"}
Nov 02 07:34:59.015765 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:59.015730    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:59.016289 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:59.016190    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:34:59.017296 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:34:59.017274    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:35:02.118558 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:02.118513    4898 generic.go:356] "Generic (PLEG): container finished" podID="8a892982-23b0-4909-b72b-4822102991ea" containerID="f6182d5f82b05e791fd16ffad4bbbd47eed5d6a85f292b0faf289f240ede13ec" exitCode=0
Nov 02 07:35:02.119084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:02.118597    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8" event={"ID":"8a892982-23b0-4909-b72b-4822102991ea","Type":"ContainerDied","Data":"f6182d5f82b05e791fd16ffad4bbbd47eed5d6a85f292b0faf289f240ede13ec"}
Nov 02 07:35:02.120682 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:02.120649    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2" event={"ID":"7a4c197d-accd-4564-a8fc-a2b2e1958846","Type":"ContainerStarted","Data":"5141b5f628131396d02bf7acd7e65f2dc800b599940850ce07203b5fc18c4c65"}
Nov 02 07:35:02.211470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:02.211413    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2" podStartSLOduration=1.898579026 podStartE2EDuration="6.211399538s" totalImagesPullingTime="4.312820512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:02.209238647 +0000 UTC m=+652.899565610" watchObservedRunningTime="2026-11-02 07:35:02.211399538 +0000 UTC m=+652.901726503"
Nov 02 07:35:03.128745 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:03.128710    4898 generic.go:356] "Generic (PLEG): container finished" podID="7a4c197d-accd-4564-a8fc-a2b2e1958846" containerID="5141b5f628131396d02bf7acd7e65f2dc800b599940850ce07203b5fc18c4c65" exitCode=0
Nov 02 07:35:03.129257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:03.128804    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2" event={"ID":"7a4c197d-accd-4564-a8fc-a2b2e1958846","Type":"ContainerDied","Data":"5141b5f628131396d02bf7acd7e65f2dc800b599940850ce07203b5fc18c4c65"}
Nov 02 07:35:03.395520 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:03.395440    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:35:03.401723 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:03.401690    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a892982-23b0-4909-b72b-4822102991ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a892982-23b0-4909-b72b-4822102991ea-maas-api-xks-ca\") pod \"8a892982-23b0-4909-b72b-4822102991ea\" (UID: \"8a892982-23b0-4909-b72b-4822102991ea\") "
Nov 02 07:35:03.401852 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:03.401764    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a892982-23b0-4909-b72b-4822102991ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a892982-23b0-4909-b72b-4822102991ea-maas-api-openshift-ca\") pod \"8a892982-23b0-4909-b72b-4822102991ea\" (UID: \"8a892982-23b0-4909-b72b-4822102991ea\") "
Nov 02 07:35:03.401852 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:03.401818    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a892982-23b0-4909-b72b-4822102991ea-kube-api-access-jchn5\" (UniqueName: \"kubernetes.io/projected/8a892982-23b0-4909-b72b-4822102991ea-kube-api-access-jchn5\") pod \"8a892982-23b0-4909-b72b-4822102991ea\" (UID: \"8a892982-23b0-4909-b72b-4822102991ea\") "
Nov 02 07:35:03.402738 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:03.402688    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a892982-23b0-4909-b72b-4822102991ea-maas-api-openshift-ca" pod "8a892982-23b0-4909-b72b-4822102991ea" (UID: "8a892982-23b0-4909-b72b-4822102991ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:03.405203 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:03.405177    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a892982-23b0-4909-b72b-4822102991ea-kube-api-access-jchn5" pod "8a892982-23b0-4909-b72b-4822102991ea" (UID: "8a892982-23b0-4909-b72b-4822102991ea"). InnerVolumeSpecName "kube-api-access-jchn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:03.405295 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:03.405177    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a892982-23b0-4909-b72b-4822102991ea-maas-api-xks-ca" pod "8a892982-23b0-4909-b72b-4822102991ea" (UID: "8a892982-23b0-4909-b72b-4822102991ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:35:03.504169 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:03.503717    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a892982-23b0-4909-b72b-4822102991ea-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:03.504169 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:03.503754    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a892982-23b0-4909-b72b-4822102991ea-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:03.504169 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:03.503769    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jchn5\" (UniqueName: \"kubernetes.io/projected/8a892982-23b0-4909-b72b-4822102991ea-kube-api-access-jchn5\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:04.137072 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.137033    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8" event={"ID":"8a892982-23b0-4909-b72b-4822102991ea","Type":"ContainerDied","Data":"7fb17c921ca86dc0abf5f5d11b6b2fb403bdb2f64c368d385815b0feef01a1fc"}
Nov 02 07:35:04.137468 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.137098    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb17c921ca86dc0abf5f5d11b6b2fb403bdb2f64c368d385815b0feef01a1fc"
Nov 02 07:35:04.137468 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.137052    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"
Nov 02 07:35:04.345033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.345004    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:35:04.414552 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.414521    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-xks-ca\") pod \"7a4c197d-accd-4564-a8fc-a2b2e1958846\" (UID: \"7a4c197d-accd-4564-a8fc-a2b2e1958846\") "
Nov 02 07:35:04.414778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.414560    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a4c197d-accd-4564-a8fc-a2b2e1958846-kube-api-access-fvlvk\" (UniqueName: \"kubernetes.io/projected/7a4c197d-accd-4564-a8fc-a2b2e1958846-kube-api-access-fvlvk\") pod \"7a4c197d-accd-4564-a8fc-a2b2e1958846\" (UID: \"7a4c197d-accd-4564-a8fc-a2b2e1958846\") "
Nov 02 07:35:04.414778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.414676    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-openshift-ca\") pod \"7a4c197d-accd-4564-a8fc-a2b2e1958846\" (UID: \"7a4c197d-accd-4564-a8fc-a2b2e1958846\") "
Nov 02 07:35:04.415394 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.415344    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-openshift-ca" pod "7a4c197d-accd-4564-a8fc-a2b2e1958846" (UID: "7a4c197d-accd-4564-a8fc-a2b2e1958846"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:04.416621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.416598    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-xks-ca" pod "7a4c197d-accd-4564-a8fc-a2b2e1958846" (UID: "7a4c197d-accd-4564-a8fc-a2b2e1958846"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:35:04.417119 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.417104    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4c197d-accd-4564-a8fc-a2b2e1958846-kube-api-access-fvlvk" pod "7a4c197d-accd-4564-a8fc-a2b2e1958846" (UID: "7a4c197d-accd-4564-a8fc-a2b2e1958846"). InnerVolumeSpecName "kube-api-access-fvlvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:04.515975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.515881    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:04.515975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.515942    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvlvk\" (UniqueName: \"kubernetes.io/projected/7a4c197d-accd-4564-a8fc-a2b2e1958846-kube-api-access-fvlvk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:04.515975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.515954    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a4c197d-accd-4564-a8fc-a2b2e1958846-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:04.732299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.730589    4898 patch_prober.go:29] interesting pod/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:57430->10.132.0.54:15021: read: connection reset by peer" start-of-body=
Nov 02 07:35:04.732299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:04.730687    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:57430->10.132.0.54:15021: read: connection reset by peer"
Nov 02 07:35:05.144347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.144256    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"
Nov 02 07:35:05.144347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.144285    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2" event={"ID":"7a4c197d-accd-4564-a8fc-a2b2e1958846","Type":"ContainerDied","Data":"3deb8d9637db3c8908b8a96f5aaf92195252ebdb5b7c08c5d28a635c42febd76"}
Nov 02 07:35:05.144347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.144345    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3deb8d9637db3c8908b8a96f5aaf92195252ebdb5b7c08c5d28a635c42febd76"
Nov 02 07:35:05.146090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.146067    4898 generic.go:356] "Generic (PLEG): container finished" podID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerID="0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016" exitCode=0
Nov 02 07:35:05.146202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.146137    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" event={"ID":"569b59f3-7992-4441-a070-3efb6bf36cb6","Type":"ContainerDied","Data":"0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016"}
Nov 02 07:35:05.146504 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.146488    4898 scope.go:122] "RemoveContainer" containerID="0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016"
Nov 02 07:35:05.147790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.147757    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:05.148111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.147845    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:05.148111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.147910    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:05.979469 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.979432    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"]
Nov 02 07:35:05.980471 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.980440    4898 cpu_manager.go:414] "containerMap: removing container" podUID="8a892982-23b0-4909-b72b-4822102991ea" containerName="revoke-keys"
Nov 02 07:35:05.980644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.980629    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a892982-23b0-4909-b72b-4822102991ea" containerName="revoke-keys"
Nov 02 07:35:05.980762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.980751    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7a4c197d-accd-4564-a8fc-a2b2e1958846" containerName="revoke-keys"
Nov 02 07:35:05.980849 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.980838    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a4c197d-accd-4564-a8fc-a2b2e1958846" containerName="revoke-keys"
Nov 02 07:35:05.981086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.981072    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a4c197d-accd-4564-a8fc-a2b2e1958846" containerName="revoke-keys"
Nov 02 07:35:05.981191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:05.981181    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a892982-23b0-4909-b72b-4822102991ea" containerName="revoke-keys"
Nov 02 07:35:06.030878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.030371    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"]
Nov 02 07:35:06.030878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.030567    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:06.034686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.034647    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2\" (UID: \"e6ee2b53-f403-4494-a33e-dd8a2941f792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:06.034847 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.034732    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gcn\" (UniqueName: \"kubernetes.io/projected/e6ee2b53-f403-4494-a33e-dd8a2941f792-kube-api-access-f4gcn\") pod \"maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2\" (UID: \"e6ee2b53-f403-4494-a33e-dd8a2941f792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:06.034847 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.034761    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2\" (UID: \"e6ee2b53-f403-4494-a33e-dd8a2941f792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:06.047394 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.047124    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:35:06.060912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.060199    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:35:06.061315 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.061295    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:35:06.061428 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.061337    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:35:06.136022 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.135974    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gcn\" (UniqueName: \"kubernetes.io/projected/e6ee2b53-f403-4494-a33e-dd8a2941f792-kube-api-access-f4gcn\") pod \"maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2\" (UID: \"e6ee2b53-f403-4494-a33e-dd8a2941f792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:06.136022 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.136025    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2\" (UID: \"e6ee2b53-f403-4494-a33e-dd8a2941f792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:06.136292 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.136175    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2\" (UID: \"e6ee2b53-f403-4494-a33e-dd8a2941f792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:06.137652 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.137623    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2\" (UID: \"e6ee2b53-f403-4494-a33e-dd8a2941f792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:06.138907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.138870    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2\" (UID: \"e6ee2b53-f403-4494-a33e-dd8a2941f792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:06.156652 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.155669    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" event={"ID":"569b59f3-7992-4441-a070-3efb6bf36cb6","Type":"ContainerStarted","Data":"8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb"}
Nov 02 07:35:06.180915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.175753    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gcn\" (UniqueName: \"kubernetes.io/projected/e6ee2b53-f403-4494-a33e-dd8a2941f792-kube-api-access-f4gcn\") pod \"maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2\" (UID: \"e6ee2b53-f403-4494-a33e-dd8a2941f792\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:06.350771 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.350687    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:06.694027 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:35:06.693991    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ee2b53_f403_4494_a33e_dd8a2941f792.slice/crio-08877a40e939ce5285f57716ec3318c0f1e9c2facc5b0b997a059c3a923449a8 WatchSource:0}: Error finding container 08877a40e939ce5285f57716ec3318c0f1e9c2facc5b0b997a059c3a923449a8: Status 404 returned error can't find the container with id 08877a40e939ce5285f57716ec3318c0f1e9c2facc5b0b997a059c3a923449a8
Nov 02 07:35:06.694240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.694216    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"]
Nov 02 07:35:06.748248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.748220    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:35:06.748441 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.748418    4898 patch_prober.go:29] interesting pod/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 07:35:06.748496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.748473    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 07:35:06.912574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:06.912545    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"]
Nov 02 07:35:07.194306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.194032    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2" event={"ID":"e6ee2b53-f403-4494-a33e-dd8a2941f792","Type":"ContainerStarted","Data":"cd7feeb9a09237fb5e45580d00c7b7fa0b82fac1b4096ce507bd4ffaf84ade44"}
Nov 02 07:35:07.194306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.194089    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2" event={"ID":"e6ee2b53-f403-4494-a33e-dd8a2941f792","Type":"ContainerStarted","Data":"08877a40e939ce5285f57716ec3318c0f1e9c2facc5b0b997a059c3a923449a8"}
Nov 02 07:35:07.238022 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.237961    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2" podStartSLOduration=2.237941382 podStartE2EDuration="2.237941382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:07.235336408 +0000 UTC m=+657.925663369" watchObservedRunningTime="2026-11-02 07:35:07.237941382 +0000 UTC m=+657.928268347"
Nov 02 07:35:07.520147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.520110    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"]
Nov 02 07:35:07.525073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.525042    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:07.527312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.527279    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"]
Nov 02 07:35:07.654481 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.654445    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb\" (UID: \"525f3483-1ccf-4e00-8453-e1be717300c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:07.654683 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.654647    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slp9z\" (UniqueName: \"kubernetes.io/projected/525f3483-1ccf-4e00-8453-e1be717300c3-kube-api-access-slp9z\") pod \"maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb\" (UID: \"525f3483-1ccf-4e00-8453-e1be717300c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:07.654761 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.654695    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb\" (UID: \"525f3483-1ccf-4e00-8453-e1be717300c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:07.748100 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.748057    4898 patch_prober.go:29] interesting pod/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 07:35:07.748269 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.748118    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 07:35:07.760915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.757483    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb\" (UID: \"525f3483-1ccf-4e00-8453-e1be717300c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:07.760915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.757578    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slp9z\" (UniqueName: \"kubernetes.io/projected/525f3483-1ccf-4e00-8453-e1be717300c3-kube-api-access-slp9z\") pod \"maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb\" (UID: \"525f3483-1ccf-4e00-8453-e1be717300c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:07.760915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.757615    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb\" (UID: \"525f3483-1ccf-4e00-8453-e1be717300c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:07.760915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.758453    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb\" (UID: \"525f3483-1ccf-4e00-8453-e1be717300c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:07.782355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.782323    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb\" (UID: \"525f3483-1ccf-4e00-8453-e1be717300c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:07.788075 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.788040    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slp9z\" (UniqueName: \"kubernetes.io/projected/525f3483-1ccf-4e00-8453-e1be717300c3-kube-api-access-slp9z\") pod \"maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb\" (UID: \"525f3483-1ccf-4e00-8453-e1be717300c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:07.877330 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:07.877067    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:08.182914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:08.182846    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"]
Nov 02 07:35:08.187362 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:35:08.187326    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525f3483_1ccf_4e00_8453_e1be717300c3.slice/crio-0fdcdf58f26d4d43c452981c65685e63946eb3f438d4ae54a1b354a738b6d2c0 WatchSource:0}: Error finding container 0fdcdf58f26d4d43c452981c65685e63946eb3f438d4ae54a1b354a738b6d2c0: Status 404 returned error can't find the container with id 0fdcdf58f26d4d43c452981c65685e63946eb3f438d4ae54a1b354a738b6d2c0
Nov 02 07:35:08.200454 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:08.200426    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb" event={"ID":"525f3483-1ccf-4e00-8453-e1be717300c3","Type":"ContainerStarted","Data":"0fdcdf58f26d4d43c452981c65685e63946eb3f438d4ae54a1b354a738b6d2c0"}
Nov 02 07:35:08.202319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:08.202228    4898 generic.go:356] "Generic (PLEG): container finished" podID="e6ee2b53-f403-4494-a33e-dd8a2941f792" containerID="cd7feeb9a09237fb5e45580d00c7b7fa0b82fac1b4096ce507bd4ffaf84ade44" exitCode=0
Nov 02 07:35:08.202414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:08.202313    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2" event={"ID":"e6ee2b53-f403-4494-a33e-dd8a2941f792","Type":"ContainerDied","Data":"cd7feeb9a09237fb5e45580d00c7b7fa0b82fac1b4096ce507bd4ffaf84ade44"}
Nov 02 07:35:08.749477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:08.749433    4898 patch_prober.go:29] interesting pod/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 07:35:08.749653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:08.749507    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 07:35:09.210725 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.210688    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb" event={"ID":"525f3483-1ccf-4e00-8453-e1be717300c3","Type":"ContainerStarted","Data":"6e5c5ff508ee19d984056b1572b924fce2614853bba37c67d69baa50ea2fd4dd"}
Nov 02 07:35:09.254531 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.254470    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb" podStartSLOduration=2.25445779 podStartE2EDuration="2.25445779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:09.253826035 +0000 UTC m=+659.944152996" watchObservedRunningTime="2026-11-02 07:35:09.25445779 +0000 UTC m=+659.944784754"
Nov 02 07:35:09.423311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.423286    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:09.506473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.495109    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-openshift-ca\") pod \"e6ee2b53-f403-4494-a33e-dd8a2941f792\" (UID: \"e6ee2b53-f403-4494-a33e-dd8a2941f792\") "
Nov 02 07:35:09.506473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.495535    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-openshift-ca" pod "e6ee2b53-f403-4494-a33e-dd8a2941f792" (UID: "e6ee2b53-f403-4494-a33e-dd8a2941f792"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:09.506473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.495600    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ee2b53-f403-4494-a33e-dd8a2941f792-kube-api-access-f4gcn\" (UniqueName: \"kubernetes.io/projected/e6ee2b53-f403-4494-a33e-dd8a2941f792-kube-api-access-f4gcn\") pod \"e6ee2b53-f403-4494-a33e-dd8a2941f792\" (UID: \"e6ee2b53-f403-4494-a33e-dd8a2941f792\") "
Nov 02 07:35:09.506473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.495968    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-xks-ca\") pod \"e6ee2b53-f403-4494-a33e-dd8a2941f792\" (UID: \"e6ee2b53-f403-4494-a33e-dd8a2941f792\") "
Nov 02 07:35:09.506473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.496712    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:09.506473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.504334    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-xks-ca" pod "e6ee2b53-f403-4494-a33e-dd8a2941f792" (UID: "e6ee2b53-f403-4494-a33e-dd8a2941f792"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:35:09.506473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.505364    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ee2b53-f403-4494-a33e-dd8a2941f792-kube-api-access-f4gcn" pod "e6ee2b53-f403-4494-a33e-dd8a2941f792" (UID: "e6ee2b53-f403-4494-a33e-dd8a2941f792"). InnerVolumeSpecName "kube-api-access-f4gcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:09.597948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.597916    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4gcn\" (UniqueName: \"kubernetes.io/projected/e6ee2b53-f403-4494-a33e-dd8a2941f792-kube-api-access-f4gcn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:09.597948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.597940    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ee2b53-f403-4494-a33e-dd8a2941f792-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:09.747902 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.747849    4898 patch_prober.go:29] interesting pod/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 07:35:09.748080 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:09.747933    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 07:35:10.218370 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.218338    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2" event={"ID":"e6ee2b53-f403-4494-a33e-dd8a2941f792","Type":"ContainerDied","Data":"08877a40e939ce5285f57716ec3318c0f1e9c2facc5b0b997a059c3a923449a8"}
Nov 02 07:35:10.218783 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.218384    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08877a40e939ce5285f57716ec3318c0f1e9c2facc5b0b997a059c3a923449a8"
Nov 02 07:35:10.218783 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.218446    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"
Nov 02 07:35:10.220946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.220919    4898 generic.go:356] "Generic (PLEG): container finished" podID="525f3483-1ccf-4e00-8453-e1be717300c3" containerID="6e5c5ff508ee19d984056b1572b924fce2614853bba37c67d69baa50ea2fd4dd" exitCode=0
Nov 02 07:35:10.221086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.221012    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb" event={"ID":"525f3483-1ccf-4e00-8453-e1be717300c3","Type":"ContainerDied","Data":"6e5c5ff508ee19d984056b1572b924fce2614853bba37c67d69baa50ea2fd4dd"}
Nov 02 07:35:10.222932 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.222913    4898 generic.go:356] "Generic (PLEG): container finished" podID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerID="cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550" exitCode=0
Nov 02 07:35:10.223021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.222970    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" event={"ID":"c5445332-4238-4877-a4d1-073fa1cb3bbd","Type":"ContainerDied","Data":"cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550"}
Nov 02 07:35:10.223280 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.223267    4898 scope.go:122] "RemoveContainer" containerID="cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550"
Nov 02 07:35:10.224705 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.224673    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:10.224786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.224768    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:10.224832 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.224817    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:10.749463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.749033    4898 patch_prober.go:29] interesting pod/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 07:35:10.749463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:10.749099    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 07:35:11.010851 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.010756    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"]
Nov 02 07:35:11.231304 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.231259    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" event={"ID":"c5445332-4238-4877-a4d1-073fa1cb3bbd","Type":"ContainerStarted","Data":"cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580"}
Nov 02 07:35:11.428079 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.428055    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:11.492661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.492618    4898 patch_prober.go:29] interesting pod/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 02 07:35:11.492847 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.492678    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 02 07:35:11.508943 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.504057    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:35:11.520129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.520100    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-openshift-ca\") pod \"525f3483-1ccf-4e00-8453-e1be717300c3\" (UID: \"525f3483-1ccf-4e00-8453-e1be717300c3\") "
Nov 02 07:35:11.520295 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.520148    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-xks-ca\") pod \"525f3483-1ccf-4e00-8453-e1be717300c3\" (UID: \"525f3483-1ccf-4e00-8453-e1be717300c3\") "
Nov 02 07:35:11.520295 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.520275    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525f3483-1ccf-4e00-8453-e1be717300c3-kube-api-access-slp9z\" (UniqueName: \"kubernetes.io/projected/525f3483-1ccf-4e00-8453-e1be717300c3-kube-api-access-slp9z\") pod \"525f3483-1ccf-4e00-8453-e1be717300c3\" (UID: \"525f3483-1ccf-4e00-8453-e1be717300c3\") "
Nov 02 07:35:11.520836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.520805    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-openshift-ca" pod "525f3483-1ccf-4e00-8453-e1be717300c3" (UID: "525f3483-1ccf-4e00-8453-e1be717300c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:11.523290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.523263    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-xks-ca" pod "525f3483-1ccf-4e00-8453-e1be717300c3" (UID: "525f3483-1ccf-4e00-8453-e1be717300c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:35:11.526588 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.526564    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525f3483-1ccf-4e00-8453-e1be717300c3-kube-api-access-slp9z" pod "525f3483-1ccf-4e00-8453-e1be717300c3" (UID: "525f3483-1ccf-4e00-8453-e1be717300c3"). InnerVolumeSpecName "kube-api-access-slp9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:11.621561 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.621482    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:11.621561 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.621511    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525f3483-1ccf-4e00-8453-e1be717300c3-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:11.621561 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.621521    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slp9z\" (UniqueName: \"kubernetes.io/projected/525f3483-1ccf-4e00-8453-e1be717300c3-kube-api-access-slp9z\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:11.789914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.774165    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:35:11.789914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.775434    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:35:11.789914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.781110    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:35:11.828917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.828867    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:35:11.914049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:11.913970    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm" podUID="507a969f-d4b9-461e-a388-e8ff376c766e" containerName="payload-pre-processing" containerID="cri-o://f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6" gracePeriod=25
Nov 02 07:35:12.196507 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.196431    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"]
Nov 02 07:35:12.200586 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.200553    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9780053-nm7t8"]
Nov 02 07:35:12.206813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.206788    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"
Nov 02 07:35:12.228169 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.228135    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507a969f-d4b9-461e-a388-e8ff376c766e-kube-api-access-npnfx\" (UniqueName: \"kubernetes.io/projected/507a969f-d4b9-461e-a388-e8ff376c766e-kube-api-access-npnfx\") pod \"507a969f-d4b9-461e-a388-e8ff376c766e\" (UID: \"507a969f-d4b9-461e-a388-e8ff376c766e\") "
Nov 02 07:35:12.228169 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.228178    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507a969f-d4b9-461e-a388-e8ff376c766e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507a969f-d4b9-461e-a388-e8ff376c766e-plugins-config-volume\") pod \"507a969f-d4b9-461e-a388-e8ff376c766e\" (UID: \"507a969f-d4b9-461e-a388-e8ff376c766e\") "
Nov 02 07:35:12.228576 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.228555    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507a969f-d4b9-461e-a388-e8ff376c766e-plugins-config-volume" pod "507a969f-d4b9-461e-a388-e8ff376c766e" (UID: "507a969f-d4b9-461e-a388-e8ff376c766e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:12.230365 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.230346    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507a969f-d4b9-461e-a388-e8ff376c766e-kube-api-access-npnfx" pod "507a969f-d4b9-461e-a388-e8ff376c766e" (UID: "507a969f-d4b9-461e-a388-e8ff376c766e"). InnerVolumeSpecName "kube-api-access-npnfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:12.238009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.237989    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"
Nov 02 07:35:12.238381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.237984    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb" event={"ID":"525f3483-1ccf-4e00-8453-e1be717300c3","Type":"ContainerDied","Data":"0fdcdf58f26d4d43c452981c65685e63946eb3f438d4ae54a1b354a738b6d2c0"}
Nov 02 07:35:12.238381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.238119    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fdcdf58f26d4d43c452981c65685e63946eb3f438d4ae54a1b354a738b6d2c0"
Nov 02 07:35:12.239340 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.239317    4898 generic.go:356] "Generic (PLEG): container finished" podID="507a969f-d4b9-461e-a388-e8ff376c766e" containerID="f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6" exitCode=0
Nov 02 07:35:12.239443 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.239357    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"
Nov 02 07:35:12.239443 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.239398    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm" event={"ID":"507a969f-d4b9-461e-a388-e8ff376c766e","Type":"ContainerDied","Data":"f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6"}
Nov 02 07:35:12.239559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.239442    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm" event={"ID":"507a969f-d4b9-461e-a388-e8ff376c766e","Type":"ContainerDied","Data":"41b88368c3b3467dc60a4392148901890ac0d3cfe325e1d9563096126d78e7e4"}
Nov 02 07:35:12.239559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.239456    4898 scope.go:122] "RemoveContainer" containerID="f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6"
Nov 02 07:35:12.249388 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.249332    4898 scope.go:122] "RemoveContainer" containerID="f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6"
Nov 02 07:35:12.249697 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:35:12.249649    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6\": container with ID starting with f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6 not found: ID does not exist" containerID="f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6"
Nov 02 07:35:12.249697 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.249686    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6"} err="failed to get container status \"f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6\": rpc error: code = NotFound desc = could not find container \"f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6\": container with ID starting with f878bc516a7036bd63af6b4a881943d20fe34960682c517fdead161509f8bcb6 not found: ID does not exist"
Nov 02 07:35:12.337438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.332024    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npnfx\" (UniqueName: \"kubernetes.io/projected/507a969f-d4b9-461e-a388-e8ff376c766e-kube-api-access-npnfx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:12.337438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.332074    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507a969f-d4b9-461e-a388-e8ff376c766e-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:12.337438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.335024    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"]
Nov 02 07:35:12.337438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.335380    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy" containerID="cri-o://1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763" gracePeriod=30
Nov 02 07:35:12.342982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.342951    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"]
Nov 02 07:35:12.347553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.347527    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ff8e809-cb5877c5-r2twm"]
Nov 02 07:35:12.482527 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.482444    4898 patch_prober.go:29] interesting pod/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 02 07:35:12.482681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.482523    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 02 07:35:12.779967 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.779562    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"]
Nov 02 07:35:12.783693 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:12.783667    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8ff8e809-lwcj2"]
Nov 02 07:35:13.482147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.482115    4898 patch_prober.go:29] interesting pod/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 02 07:35:13.482958 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.482181    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 02 07:35:13.484104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.484078    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507a969f-d4b9-461e-a388-e8ff376c766e" path="/var/lib/kubelet/pods/507a969f-d4b9-461e-a388-e8ff376c766e/volumes"
Nov 02 07:35:13.484690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.484668    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a4c197d-accd-4564-a8fc-a2b2e1958846" path="/var/lib/kubelet/pods/7a4c197d-accd-4564-a8fc-a2b2e1958846/volumes"
Nov 02 07:35:13.485191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.485173    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a892982-23b0-4909-b72b-4822102991ea" path="/var/lib/kubelet/pods/8a892982-23b0-4909-b72b-4822102991ea/volumes"
Nov 02 07:35:13.588982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.587769    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"]
Nov 02 07:35:13.588982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.588284    4898 cpu_manager.go:414] "containerMap: removing container" podUID="507a969f-d4b9-461e-a388-e8ff376c766e" containerName="payload-pre-processing"
Nov 02 07:35:13.588982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.588299    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507a969f-d4b9-461e-a388-e8ff376c766e" containerName="payload-pre-processing"
Nov 02 07:35:13.588982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.588333    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e6ee2b53-f403-4494-a33e-dd8a2941f792" containerName="revoke-keys"
Nov 02 07:35:13.588982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.588339    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ee2b53-f403-4494-a33e-dd8a2941f792" containerName="revoke-keys"
Nov 02 07:35:13.588982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.588348    4898 cpu_manager.go:414] "containerMap: removing container" podUID="525f3483-1ccf-4e00-8453-e1be717300c3" containerName="revoke-keys"
Nov 02 07:35:13.588982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.588353    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525f3483-1ccf-4e00-8453-e1be717300c3" containerName="revoke-keys"
Nov 02 07:35:13.588982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.588422    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ee2b53-f403-4494-a33e-dd8a2941f792" containerName="revoke-keys"
Nov 02 07:35:13.588982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.588431    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="525f3483-1ccf-4e00-8453-e1be717300c3" containerName="revoke-keys"
Nov 02 07:35:13.588982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.588439    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="507a969f-d4b9-461e-a388-e8ff376c766e" containerName="payload-pre-processing"
Nov 02 07:35:13.593271 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.593249    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"]
Nov 02 07:35:13.593382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.593355    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.612702 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.612677    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-354660-openshift-default-dockercfg-kgbn5\""
Nov 02 07:35:13.645466 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.645438    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53de2d77-31bc-415c-8c29-d768438375af-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.645583 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.645499    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.645583 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.645561    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53de2d77-31bc-415c-8c29-d768438375af-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.645670 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.645595    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.645670 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.645614    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.645739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.645705    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.645770 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.645752    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkqbk\" (UniqueName: \"kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-kube-api-access-qkqbk\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.645803 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.645787    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.645845 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.645831    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.747085 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.746991    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.747085 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.747050    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53de2d77-31bc-415c-8c29-d768438375af-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.747302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.747120    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.747302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.747167    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.747302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.747254    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.747462 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.747323    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkqbk\" (UniqueName: \"kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-kube-api-access-qkqbk\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.747462 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.747392    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.747462 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.747444    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.747610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.747519    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53de2d77-31bc-415c-8c29-d768438375af-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.747670 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.747634    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.747909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.747872    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.748019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.747923    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.748019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.747964    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.748401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.748380    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53de2d77-31bc-415c-8c29-d768438375af-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.749981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.749954    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.750090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.750004    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53de2d77-31bc-415c-8c29-d768438375af-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.789466 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.789414    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.790812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.790769    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkqbk\" (UniqueName: \"kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-kube-api-access-qkqbk\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:13.916309 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:13.916270    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:14.018475 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.018431    4898 patch_prober.go:29] interesting pod/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:35:14.018640 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.018492    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:35:14.253324 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:35:14.253293    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53de2d77_31bc_415c_8c29_d768438375af.slice/crio-8359db1f22a3c88efd9cafd4fa3a7d8875aa26cf75bf390ca52438071d96c922 WatchSource:0}: Error finding container 8359db1f22a3c88efd9cafd4fa3a7d8875aa26cf75bf390ca52438071d96c922: Status 404 returned error can't find the container with id 8359db1f22a3c88efd9cafd4fa3a7d8875aa26cf75bf390ca52438071d96c922
Nov 02 07:35:14.257927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.254940    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"]
Nov 02 07:35:14.257927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.255588    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:14.257927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.255662    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:14.257927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.255704    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:14.482913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.482533    4898 patch_prober.go:29] interesting pod/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 02 07:35:14.482913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.482599    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 02 07:35:14.590424 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.590340    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"]
Nov 02 07:35:14.595379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.595356    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.625748 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.625714    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0fbc17-openshift-default-dockercfg-p76f6\""
Nov 02 07:35:14.627732 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.627696    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"]
Nov 02 07:35:14.694511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.694468    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.694511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.694513    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfvvk\" (UniqueName: \"kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-kube-api-access-mfvvk\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.694756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.694539    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.694756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.694554    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.694756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.694584    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.694756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.694600    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.694756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.694636    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.694756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.694679    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.694756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.694699    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.796915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.796662    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.796915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.796718    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfvvk\" (UniqueName: \"kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-kube-api-access-mfvvk\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.796915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.796763    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.796915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.796788    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.796915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.796835    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.796915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.796861    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.797369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.796933    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.797369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.797004    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.797369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.797033    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.797538 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.797508    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.800912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.797755    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.800912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.798853    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.800912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.799494    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.800912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.799730    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.828934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.821839    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.828934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.823545    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfvvk\" (UniqueName: \"kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-kube-api-access-mfvvk\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.831421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.831392    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.834558 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.833716    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:14.913970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:14.913867    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:15.017808 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:35:15.017355    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:35:15.276431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:15.276396    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"]
Nov 02 07:35:15.292041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:15.291601    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:15.292041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:15.291675    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:15.292041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:15.291702    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:15.293860 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:15.293831    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" event={"ID":"53de2d77-31bc-415c-8c29-d768438375af","Type":"ContainerStarted","Data":"dc11653148133e1dbbce433fdadc2fca8b60c9be24672213d9897bc89c53322d"}
Nov 02 07:35:15.294006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:15.293974    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" event={"ID":"53de2d77-31bc-415c-8c29-d768438375af","Type":"ContainerStarted","Data":"8359db1f22a3c88efd9cafd4fa3a7d8875aa26cf75bf390ca52438071d96c922"}
Nov 02 07:35:15.341009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:15.340964    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" podStartSLOduration=2.340950746 podStartE2EDuration="2.340950746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:15.339297454 +0000 UTC m=+666.029624437" watchObservedRunningTime="2026-11-02 07:35:15.340950746 +0000 UTC m=+666.031277709"
Nov 02 07:35:15.487097 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:15.487061    4898 patch_prober.go:29] interesting pod/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 02 07:35:15.516999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:15.516080    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 02 07:35:15.917073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:15.917030    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:15.922545 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:15.922518    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:16.011247 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.011194    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k" podUID="b6e84d20-8952-43a9-a57c-21b2ab0f1221" containerName="payload-processing" containerID="cri-o://a5811f053fb3f52443ddcd7f793582377fa2a6264d8c2e6d80ab18db50a8e148" gracePeriod=25
Nov 02 07:35:16.301825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.301796    4898 generic.go:356] "Generic (PLEG): container finished" podID="b6e84d20-8952-43a9-a57c-21b2ab0f1221" containerID="a5811f053fb3f52443ddcd7f793582377fa2a6264d8c2e6d80ab18db50a8e148" exitCode=0
Nov 02 07:35:16.302006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.301834    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k" event={"ID":"b6e84d20-8952-43a9-a57c-21b2ab0f1221","Type":"ContainerDied","Data":"a5811f053fb3f52443ddcd7f793582377fa2a6264d8c2e6d80ab18db50a8e148"}
Nov 02 07:35:16.303756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.303722    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" event={"ID":"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e","Type":"ContainerStarted","Data":"dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d"}
Nov 02 07:35:16.303869 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.303852    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:16.303974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.303932    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" event={"ID":"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e","Type":"ContainerStarted","Data":"1ff2fde2dbbf99dcf7be1bacf5ee6f6db70dcdc5e18051d4ceeb6e990f659c02"}
Nov 02 07:35:16.304884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.304868    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:16.334928 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.334881    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"
Nov 02 07:35:16.372729 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.372666    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" podStartSLOduration=2.372648371 podStartE2EDuration="2.372648371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:16.371409501 +0000 UTC m=+667.061736517" watchObservedRunningTime="2026-11-02 07:35:16.372648371 +0000 UTC m=+667.062975338"
Nov 02 07:35:16.416372 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.416334    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e84d20-8952-43a9-a57c-21b2ab0f1221-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e84d20-8952-43a9-a57c-21b2ab0f1221-plugins-config-volume\") pod \"b6e84d20-8952-43a9-a57c-21b2ab0f1221\" (UID: \"b6e84d20-8952-43a9-a57c-21b2ab0f1221\") "
Nov 02 07:35:16.416530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.416399    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e84d20-8952-43a9-a57c-21b2ab0f1221-kube-api-access-hfr5w\" (UniqueName: \"kubernetes.io/projected/b6e84d20-8952-43a9-a57c-21b2ab0f1221-kube-api-access-hfr5w\") pod \"b6e84d20-8952-43a9-a57c-21b2ab0f1221\" (UID: \"b6e84d20-8952-43a9-a57c-21b2ab0f1221\") "
Nov 02 07:35:16.416780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.416747    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e84d20-8952-43a9-a57c-21b2ab0f1221-plugins-config-volume" pod "b6e84d20-8952-43a9-a57c-21b2ab0f1221" (UID: "b6e84d20-8952-43a9-a57c-21b2ab0f1221"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:16.417065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.417048    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e84d20-8952-43a9-a57c-21b2ab0f1221-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:16.419511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.419482    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e84d20-8952-43a9-a57c-21b2ab0f1221-kube-api-access-hfr5w" pod "b6e84d20-8952-43a9-a57c-21b2ab0f1221" (UID: "b6e84d20-8952-43a9-a57c-21b2ab0f1221"). InnerVolumeSpecName "kube-api-access-hfr5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:16.518272 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.518191    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfr5w\" (UniqueName: \"kubernetes.io/projected/b6e84d20-8952-43a9-a57c-21b2ab0f1221-kube-api-access-hfr5w\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:16.535681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.535638    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:35:16.537827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.537798    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:35:16.620277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.620246    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"]
Nov 02 07:35:16.914976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.914943    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:16.920805 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:16.920780    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:17.312659 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.312569    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"
Nov 02 07:35:17.312659 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.312617    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k" event={"ID":"b6e84d20-8952-43a9-a57c-21b2ab0f1221","Type":"ContainerDied","Data":"69859a68f9c4a12ddfa74ba06eb13516b526b44b23214f609502ac5d9ba66cc4"}
Nov 02 07:35:17.312883 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.312671    4898 scope.go:122] "RemoveContainer" containerID="a5811f053fb3f52443ddcd7f793582377fa2a6264d8c2e6d80ab18db50a8e148"
Nov 02 07:35:17.316205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.315500    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:17.316574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.316554    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:17.423436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.423387    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"]
Nov 02 07:35:17.426224 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.426195    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-935d4e21-d9d897fb9-t559k"]
Nov 02 07:35:17.467007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.466973    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e84d20-8952-43a9-a57c-21b2ab0f1221" path="/var/lib/kubelet/pods/b6e84d20-8952-43a9-a57c-21b2ab0f1221/volumes"
Nov 02 07:35:17.624295 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.624269    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:35:17.739059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.739003    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-envoy\") pod \"6e779948-dde1-40c4-bbe7-7a365344f1ed\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") "
Nov 02 07:35:17.739262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.739091    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-credential-socket\") pod \"6e779948-dde1-40c4-bbe7-7a365344f1ed\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") "
Nov 02 07:35:17.739262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.739187    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e779948-dde1-40c4-bbe7-7a365344f1ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e779948-dde1-40c4-bbe7-7a365344f1ed-istiod-ca-cert\") pod \"6e779948-dde1-40c4-bbe7-7a365344f1ed\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") "
Nov 02 07:35:17.739262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.739213    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-token\" (UniqueName: \"kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-token\") pod \"6e779948-dde1-40c4-bbe7-7a365344f1ed\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") "
Nov 02 07:35:17.739262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.739241    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-socket\") pod \"6e779948-dde1-40c4-bbe7-7a365344f1ed\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") "
Nov 02 07:35:17.739485 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.739312    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-data\") pod \"6e779948-dde1-40c4-bbe7-7a365344f1ed\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") "
Nov 02 07:35:17.739485 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.739351    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-kube-api-access-bf9cj\" (UniqueName: \"kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-kube-api-access-bf9cj\") pod \"6e779948-dde1-40c4-bbe7-7a365344f1ed\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") "
Nov 02 07:35:17.739485 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.739416    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-podinfo\") pod \"6e779948-dde1-40c4-bbe7-7a365344f1ed\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") "
Nov 02 07:35:17.739642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.739485    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-certs\") pod \"6e779948-dde1-40c4-bbe7-7a365344f1ed\" (UID: \"6e779948-dde1-40c4-bbe7-7a365344f1ed\") "
Nov 02 07:35:17.739789 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.739749    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-socket" pod "6e779948-dde1-40c4-bbe7-7a365344f1ed" (UID: "6e779948-dde1-40c4-bbe7-7a365344f1ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:17.740002 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.739981    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.740093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.739960    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-data" pod "6e779948-dde1-40c4-bbe7-7a365344f1ed" (UID: "6e779948-dde1-40c4-bbe7-7a365344f1ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:17.740191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.740153    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-certs" pod "6e779948-dde1-40c4-bbe7-7a365344f1ed" (UID: "6e779948-dde1-40c4-bbe7-7a365344f1ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:17.740308 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.740285    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e779948-dde1-40c4-bbe7-7a365344f1ed-istiod-ca-cert" pod "6e779948-dde1-40c4-bbe7-7a365344f1ed" (UID: "6e779948-dde1-40c4-bbe7-7a365344f1ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:17.740556 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.740536    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-credential-socket" pod "6e779948-dde1-40c4-bbe7-7a365344f1ed" (UID: "6e779948-dde1-40c4-bbe7-7a365344f1ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:17.742810 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.742779    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-podinfo" pod "6e779948-dde1-40c4-bbe7-7a365344f1ed" (UID: "6e779948-dde1-40c4-bbe7-7a365344f1ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:35:17.743099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.743076    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-token" pod "6e779948-dde1-40c4-bbe7-7a365344f1ed" (UID: "6e779948-dde1-40c4-bbe7-7a365344f1ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:17.748969 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.746100    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-kube-api-access-bf9cj" pod "6e779948-dde1-40c4-bbe7-7a365344f1ed" (UID: "6e779948-dde1-40c4-bbe7-7a365344f1ed"). InnerVolumeSpecName "kube-api-access-bf9cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:17.749088 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.749001    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-envoy" pod "6e779948-dde1-40c4-bbe7-7a365344f1ed" (UID: "6e779948-dde1-40c4-bbe7-7a365344f1ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:17.841091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.841042    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.841091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.841085    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf9cj\" (UniqueName: \"kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-kube-api-access-bf9cj\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.841091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.841097    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.841091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.841106    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.841355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.841115    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.841355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.841125    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e779948-dde1-40c4-bbe7-7a365344f1ed-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.841355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.841133    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e779948-dde1-40c4-bbe7-7a365344f1ed-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.841355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.841141    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e779948-dde1-40c4-bbe7-7a365344f1ed-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:17.931190 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:17.931157    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"]
Nov 02 07:35:18.271962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.271872    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"]
Nov 02 07:35:18.280811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.280784    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be1fb7c9-zbzm2"]
Nov 02 07:35:18.322370 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.322332    4898 generic.go:356] "Generic (PLEG): container finished" podID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerID="1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763" exitCode=0
Nov 02 07:35:18.322525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.322389    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"
Nov 02 07:35:18.322525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.322409    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" event={"ID":"6e779948-dde1-40c4-bbe7-7a365344f1ed","Type":"ContainerDied","Data":"1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763"}
Nov 02 07:35:18.322525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.322464    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2" event={"ID":"6e779948-dde1-40c4-bbe7-7a365344f1ed","Type":"ContainerDied","Data":"ea902552b586a1081cca67a3c4236dec78ad929120183576f5b7269aea4f9c6c"}
Nov 02 07:35:18.322525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.322489    4898 scope.go:122] "RemoveContainer" containerID="1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763"
Nov 02 07:35:18.333983 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.333963    4898 scope.go:122] "RemoveContainer" containerID="c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845"
Nov 02 07:35:18.344203 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.344183    4898 scope.go:122] "RemoveContainer" containerID="1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763"
Nov 02 07:35:18.344559 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:35:18.344529    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763\": container with ID starting with 1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763 not found: ID does not exist" containerID="1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763"
Nov 02 07:35:18.344952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.344571    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763"} err="failed to get container status \"1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763\": rpc error: code = NotFound desc = could not find container \"1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763\": container with ID starting with 1f6cd5fb5e7185919ec0cc6c4c31073e73730704208a2169a1704716cafe9763 not found: ID does not exist"
Nov 02 07:35:18.344952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.344595    4898 scope.go:122] "RemoveContainer" containerID="c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845"
Nov 02 07:35:18.345129 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:35:18.345110    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845\": container with ID starting with c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845 not found: ID does not exist" containerID="c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845"
Nov 02 07:35:18.345186 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.345134    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845"} err="failed to get container status \"c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845\": rpc error: code = NotFound desc = could not find container \"c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845\": container with ID starting with c6992488690589277e424427134171e633ada71e43fead8d21973444392dc845 not found: ID does not exist"
Nov 02 07:35:18.386528 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.386502    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"]
Nov 02 07:35:18.397038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:18.397012    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b9780053-gw-openshift-default-7c556db489-fkpw2"]
Nov 02 07:35:19.466101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:19.466070    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" path="/var/lib/kubelet/pods/6e779948-dde1-40c4-bbe7-7a365344f1ed/volumes"
Nov 02 07:35:19.466595 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:19.466519    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ee2b53-f403-4494-a33e-dd8a2941f792" path="/var/lib/kubelet/pods/e6ee2b53-f403-4494-a33e-dd8a2941f792/volumes"
Nov 02 07:35:19.703417 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:19.703391    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"]
Nov 02 07:35:19.703621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:19.703596    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy" containerID="cri-o://8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb" gracePeriod=30
Nov 02 07:35:21.621299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.621260    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5" podUID="151f52fe-5ecf-417a-8f40-ebffd15ea385" containerName="payload-pre-processing" containerID="cri-o://20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc" gracePeriod=25
Nov 02 07:35:21.735942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.735856    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"]
Nov 02 07:35:21.736365 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.736351    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b6e84d20-8952-43a9-a57c-21b2ab0f1221" containerName="payload-processing"
Nov 02 07:35:21.736413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.736368    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e84d20-8952-43a9-a57c-21b2ab0f1221" containerName="payload-processing"
Nov 02 07:35:21.736413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.736388    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy"
Nov 02 07:35:21.736413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.736392    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy"
Nov 02 07:35:21.736413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.736406    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy"
Nov 02 07:35:21.736413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.736411    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy"
Nov 02 07:35:21.736587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.736491    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy"
Nov 02 07:35:21.736587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.736500    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e779948-dde1-40c4-bbe7-7a365344f1ed" containerName="istio-proxy"
Nov 02 07:35:21.736587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.736510    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e84d20-8952-43a9-a57c-21b2ab0f1221" containerName="payload-processing"
Nov 02 07:35:21.740016 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.739998    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"
Nov 02 07:35:21.765273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.765236    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"]
Nov 02 07:35:21.777199 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.777162    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-0fbc17\""
Nov 02 07:35:21.777199 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.777165    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-0fbc17-dockercfg-lkjr2\""
Nov 02 07:35:21.838658 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.836188    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"]
Nov 02 07:35:21.856817 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.856785    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"]
Nov 02 07:35:21.867225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.866216    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"
Nov 02 07:35:21.881345 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.881309    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1f4f74-4502-49dd-a54a-893b9264a0cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4\" (UID: \"ea1f4f74-4502-49dd-a54a-893b9264a0cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"
Nov 02 07:35:21.881476 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.881421    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8vf9\" (UniqueName: \"kubernetes.io/projected/ea1f4f74-4502-49dd-a54a-893b9264a0cf-kube-api-access-q8vf9\") pod \"payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4\" (UID: \"ea1f4f74-4502-49dd-a54a-893b9264a0cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"
Nov 02 07:35:21.941471 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.941437    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"
Nov 02 07:35:21.982655 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.982618    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151f52fe-5ecf-417a-8f40-ebffd15ea385-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151f52fe-5ecf-417a-8f40-ebffd15ea385-plugins-config-volume\") pod \"151f52fe-5ecf-417a-8f40-ebffd15ea385\" (UID: \"151f52fe-5ecf-417a-8f40-ebffd15ea385\") "
Nov 02 07:35:21.982799 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.982675    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151f52fe-5ecf-417a-8f40-ebffd15ea385-kube-api-access-2qwl6\" (UniqueName: \"kubernetes.io/projected/151f52fe-5ecf-417a-8f40-ebffd15ea385-kube-api-access-2qwl6\") pod \"151f52fe-5ecf-417a-8f40-ebffd15ea385\" (UID: \"151f52fe-5ecf-417a-8f40-ebffd15ea385\") "
Nov 02 07:35:21.983236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.982910    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8vf9\" (UniqueName: \"kubernetes.io/projected/ea1f4f74-4502-49dd-a54a-893b9264a0cf-kube-api-access-q8vf9\") pod \"payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4\" (UID: \"ea1f4f74-4502-49dd-a54a-893b9264a0cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"
Nov 02 07:35:21.983236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.983111    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qrpn\" (UniqueName: \"kubernetes.io/projected/c9606134-fab6-48c1-9543-b20f0cf28967-kube-api-access-2qrpn\") pod \"payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5\" (UID: \"c9606134-fab6-48c1-9543-b20f0cf28967\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"
Nov 02 07:35:21.983236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.983158    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9606134-fab6-48c1-9543-b20f0cf28967-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5\" (UID: \"c9606134-fab6-48c1-9543-b20f0cf28967\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"
Nov 02 07:35:21.983236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.983223    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1f4f74-4502-49dd-a54a-893b9264a0cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4\" (UID: \"ea1f4f74-4502-49dd-a54a-893b9264a0cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"
Nov 02 07:35:21.984102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.984077    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1f4f74-4502-49dd-a54a-893b9264a0cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4\" (UID: \"ea1f4f74-4502-49dd-a54a-893b9264a0cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"
Nov 02 07:35:21.984283 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.984255    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151f52fe-5ecf-417a-8f40-ebffd15ea385-plugins-config-volume" pod "151f52fe-5ecf-417a-8f40-ebffd15ea385" (UID: "151f52fe-5ecf-417a-8f40-ebffd15ea385"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:21.985550 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:21.985526    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151f52fe-5ecf-417a-8f40-ebffd15ea385-kube-api-access-2qwl6" pod "151f52fe-5ecf-417a-8f40-ebffd15ea385" (UID: "151f52fe-5ecf-417a-8f40-ebffd15ea385"). InnerVolumeSpecName "kube-api-access-2qwl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:22.009778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.009749    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8vf9\" (UniqueName: \"kubernetes.io/projected/ea1f4f74-4502-49dd-a54a-893b9264a0cf-kube-api-access-q8vf9\") pod \"payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4\" (UID: \"ea1f4f74-4502-49dd-a54a-893b9264a0cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"
Nov 02 07:35:22.054432 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.054407    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"
Nov 02 07:35:22.083950 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.083917    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qrpn\" (UniqueName: \"kubernetes.io/projected/c9606134-fab6-48c1-9543-b20f0cf28967-kube-api-access-2qrpn\") pod \"payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5\" (UID: \"c9606134-fab6-48c1-9543-b20f0cf28967\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"
Nov 02 07:35:22.084148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.083975    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9606134-fab6-48c1-9543-b20f0cf28967-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5\" (UID: \"c9606134-fab6-48c1-9543-b20f0cf28967\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"
Nov 02 07:35:22.084148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.084083    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151f52fe-5ecf-417a-8f40-ebffd15ea385-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:22.084148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.084100    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qwl6\" (UniqueName: \"kubernetes.io/projected/151f52fe-5ecf-417a-8f40-ebffd15ea385-kube-api-access-2qwl6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:22.084937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.084869    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9606134-fab6-48c1-9543-b20f0cf28967-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5\" (UID: \"c9606134-fab6-48c1-9543-b20f0cf28967\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"
Nov 02 07:35:22.107488 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.107463    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qrpn\" (UniqueName: \"kubernetes.io/projected/c9606134-fab6-48c1-9543-b20f0cf28967-kube-api-access-2qrpn\") pod \"payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5\" (UID: \"c9606134-fab6-48c1-9543-b20f0cf28967\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"
Nov 02 07:35:22.186614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.186580    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"
Nov 02 07:35:22.348402 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.348366    4898 generic.go:356] "Generic (PLEG): container finished" podID="151f52fe-5ecf-417a-8f40-ebffd15ea385" containerID="20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc" exitCode=0
Nov 02 07:35:22.348574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.348414    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"
Nov 02 07:35:22.348574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.348449    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5" event={"ID":"151f52fe-5ecf-417a-8f40-ebffd15ea385","Type":"ContainerDied","Data":"20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc"}
Nov 02 07:35:22.348574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.348506    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5" event={"ID":"151f52fe-5ecf-417a-8f40-ebffd15ea385","Type":"ContainerDied","Data":"c4f9f5457d2af3e5d5000c56a854b2272b0a0b872cde97d4c242fcfb6bac5927"}
Nov 02 07:35:22.348574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.348521    4898 scope.go:122] "RemoveContainer" containerID="20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc"
Nov 02 07:35:22.366047 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:35:22.366017    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1f4f74_4502_49dd_a54a_893b9264a0cf.slice/crio-65ceb3875cc4f48d8b0b4cb5fcf2dbe2cf7e9a483b7765c542a4c7517981aa9a WatchSource:0}: Error finding container 65ceb3875cc4f48d8b0b4cb5fcf2dbe2cf7e9a483b7765c542a4c7517981aa9a: Status 404 returned error can't find the container with id 65ceb3875cc4f48d8b0b4cb5fcf2dbe2cf7e9a483b7765c542a4c7517981aa9a
Nov 02 07:35:22.366047 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.366029    4898 scope.go:122] "RemoveContainer" containerID="20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc"
Nov 02 07:35:22.366389 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:35:22.366360    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc\": container with ID starting with 20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc not found: ID does not exist" containerID="20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc"
Nov 02 07:35:22.366463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.366403    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc"} err="failed to get container status \"20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc\": rpc error: code = NotFound desc = could not find container \"20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc\": container with ID starting with 20003f296e248c34d8fd2cd07c6d973f2d7cba1bcb545b6d679ac8ef759ccbcc not found: ID does not exist"
Nov 02 07:35:22.379949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.370047    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"]
Nov 02 07:35:22.405644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.405617    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"]
Nov 02 07:35:22.409685 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.409659    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-be1fb7c9-7f5ccb7df5-j4ns5"]
Nov 02 07:35:22.484454 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:35:22.484430    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9606134_fab6_48c1_9543_b20f0cf28967.slice/crio-790b37aa5ed24021e497a87dc90c3f8295326b0c0c949f06524e21e59561d604 WatchSource:0}: Error finding container 790b37aa5ed24021e497a87dc90c3f8295326b0c0c949f06524e21e59561d604: Status 404 returned error can't find the container with id 790b37aa5ed24021e497a87dc90c3f8295326b0c0c949f06524e21e59561d604
Nov 02 07:35:22.484554 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.484474    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"]
Nov 02 07:35:22.932442 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:22.932375    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx" podUID="fd3e3dc4-09fe-497e-92a2-21297c58aa1c" containerName="payload-pre-processing" containerID="cri-o://ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390" gracePeriod=25
Nov 02 07:35:23.190379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.190309    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"
Nov 02 07:35:23.303812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.303775    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-plugins-config-volume\") pod \"fd3e3dc4-09fe-497e-92a2-21297c58aa1c\" (UID: \"fd3e3dc4-09fe-497e-92a2-21297c58aa1c\") "
Nov 02 07:35:23.303989 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.303956    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-kube-api-access-l5hnr\" (UniqueName: \"kubernetes.io/projected/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-kube-api-access-l5hnr\") pod \"fd3e3dc4-09fe-497e-92a2-21297c58aa1c\" (UID: \"fd3e3dc4-09fe-497e-92a2-21297c58aa1c\") "
Nov 02 07:35:23.304229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.304202    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-plugins-config-volume" pod "fd3e3dc4-09fe-497e-92a2-21297c58aa1c" (UID: "fd3e3dc4-09fe-497e-92a2-21297c58aa1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:23.304350 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.304328    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:23.306066 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.306045    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-kube-api-access-l5hnr" pod "fd3e3dc4-09fe-497e-92a2-21297c58aa1c" (UID: "fd3e3dc4-09fe-497e-92a2-21297c58aa1c"). InnerVolumeSpecName "kube-api-access-l5hnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:23.357578 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.357539    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5" event={"ID":"c9606134-fab6-48c1-9543-b20f0cf28967","Type":"ContainerStarted","Data":"e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24"}
Nov 02 07:35:23.357780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.357604    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5" event={"ID":"c9606134-fab6-48c1-9543-b20f0cf28967","Type":"ContainerStarted","Data":"790b37aa5ed24021e497a87dc90c3f8295326b0c0c949f06524e21e59561d604"}
Nov 02 07:35:23.357780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.357636    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"
Nov 02 07:35:23.359839 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.359695    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4" event={"ID":"ea1f4f74-4502-49dd-a54a-893b9264a0cf","Type":"ContainerStarted","Data":"1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8"}
Nov 02 07:35:23.359839 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.359755    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4" event={"ID":"ea1f4f74-4502-49dd-a54a-893b9264a0cf","Type":"ContainerStarted","Data":"65ceb3875cc4f48d8b0b4cb5fcf2dbe2cf7e9a483b7765c542a4c7517981aa9a"}
Nov 02 07:35:23.359839 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.359782    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"
Nov 02 07:35:23.361521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.361491    4898 generic.go:356] "Generic (PLEG): container finished" podID="fd3e3dc4-09fe-497e-92a2-21297c58aa1c" containerID="ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390" exitCode=0
Nov 02 07:35:23.361648 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.361533    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx" event={"ID":"fd3e3dc4-09fe-497e-92a2-21297c58aa1c","Type":"ContainerDied","Data":"ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390"}
Nov 02 07:35:23.361648 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.361597    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx" event={"ID":"fd3e3dc4-09fe-497e-92a2-21297c58aa1c","Type":"ContainerDied","Data":"807e63965db3ba975dd2d88364c4e1638bc757a4211b6de8908882a612e73fd5"}
Nov 02 07:35:23.361648 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.361630    4898 scope.go:122] "RemoveContainer" containerID="ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390"
Nov 02 07:35:23.361811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.361801    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"
Nov 02 07:35:23.374703 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.374679    4898 scope.go:122] "RemoveContainer" containerID="ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390"
Nov 02 07:35:23.375311 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:35:23.375249    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390\": container with ID starting with ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390 not found: ID does not exist" containerID="ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390"
Nov 02 07:35:23.375311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.375285    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390"} err="failed to get container status \"ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390\": rpc error: code = NotFound desc = could not find container \"ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390\": container with ID starting with ac705887a1d3153c2a157479b3bf76ff31344037e85d1f1db899ee5d64dc3390 not found: ID does not exist"
Nov 02 07:35:23.405751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.405486    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5hnr\" (UniqueName: \"kubernetes.io/projected/fd3e3dc4-09fe-497e-92a2-21297c58aa1c-kube-api-access-l5hnr\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:23.408638 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.408598    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5" podStartSLOduration=2.408586369 podStartE2EDuration="2.408586369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:23.405688712 +0000 UTC m=+674.096015697" watchObservedRunningTime="2026-11-02 07:35:23.408586369 +0000 UTC m=+674.098913332"
Nov 02 07:35:23.446397 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.446303    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"]
Nov 02 07:35:23.451903 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.451870    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8c43c08e-cd5dbb799-2n8mx"]
Nov 02 07:35:23.463527 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.463502    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151f52fe-5ecf-417a-8f40-ebffd15ea385" path="/var/lib/kubelet/pods/151f52fe-5ecf-417a-8f40-ebffd15ea385/volumes"
Nov 02 07:35:23.473393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.473362    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3e3dc4-09fe-497e-92a2-21297c58aa1c" path="/var/lib/kubelet/pods/fd3e3dc4-09fe-497e-92a2-21297c58aa1c/volumes"
Nov 02 07:35:23.494421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:23.494371    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4" podStartSLOduration=2.494354115 podStartE2EDuration="2.494354115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:23.492436586 +0000 UTC m=+674.182763549" watchObservedRunningTime="2026-11-02 07:35:23.494354115 +0000 UTC m=+674.184681086"
Nov 02 07:35:24.756940 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:24.756911    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"]
Nov 02 07:35:24.770230 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:24.770194    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c43c08e-fltdb"]
Nov 02 07:35:24.903800 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:24.903768    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"]
Nov 02 07:35:24.904042 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:24.904017    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerName="istio-proxy" containerID="cri-o://cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580" gracePeriod=30
Nov 02 07:35:24.975119 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:24.975063    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:35:25.122254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.122212    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-certs\") pod \"569b59f3-7992-4441-a070-3efb6bf36cb6\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") "
Nov 02 07:35:25.122555 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.122513    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-token\" (UniqueName: \"kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-token\") pod \"569b59f3-7992-4441-a070-3efb6bf36cb6\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") "
Nov 02 07:35:25.122694 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.122600    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-envoy\") pod \"569b59f3-7992-4441-a070-3efb6bf36cb6\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") "
Nov 02 07:35:25.122694 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.122648    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-kube-api-access-cc9wm\" (UniqueName: \"kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-kube-api-access-cc9wm\") pod \"569b59f3-7992-4441-a070-3efb6bf36cb6\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") "
Nov 02 07:35:25.122811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.122722    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-credential-socket\") pod \"569b59f3-7992-4441-a070-3efb6bf36cb6\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") "
Nov 02 07:35:25.122871 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.122809    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569b59f3-7992-4441-a070-3efb6bf36cb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b59f3-7992-4441-a070-3efb6bf36cb6-istiod-ca-cert\") pod \"569b59f3-7992-4441-a070-3efb6bf36cb6\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") "
Nov 02 07:35:25.122871 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.122845    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-socket\") pod \"569b59f3-7992-4441-a070-3efb6bf36cb6\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") "
Nov 02 07:35:25.123033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.122908    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-data\") pod \"569b59f3-7992-4441-a070-3efb6bf36cb6\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") "
Nov 02 07:35:25.123033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.122945    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-podinfo\") pod \"569b59f3-7992-4441-a070-3efb6bf36cb6\" (UID: \"569b59f3-7992-4441-a070-3efb6bf36cb6\") "
Nov 02 07:35:25.124676 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.124640    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-certs" pod "569b59f3-7992-4441-a070-3efb6bf36cb6" (UID: "569b59f3-7992-4441-a070-3efb6bf36cb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:25.127199 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.126749    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-podinfo" pod "569b59f3-7992-4441-a070-3efb6bf36cb6" (UID: "569b59f3-7992-4441-a070-3efb6bf36cb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:35:25.127199 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.127009    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-credential-socket" pod "569b59f3-7992-4441-a070-3efb6bf36cb6" (UID: "569b59f3-7992-4441-a070-3efb6bf36cb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:25.129599 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.129004    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-envoy" pod "569b59f3-7992-4441-a070-3efb6bf36cb6" (UID: "569b59f3-7992-4441-a070-3efb6bf36cb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:25.129599 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.129331    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-data" pod "569b59f3-7992-4441-a070-3efb6bf36cb6" (UID: "569b59f3-7992-4441-a070-3efb6bf36cb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:25.129599 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.129382    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-socket" pod "569b59f3-7992-4441-a070-3efb6bf36cb6" (UID: "569b59f3-7992-4441-a070-3efb6bf36cb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:25.129599 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.129551    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-token" pod "569b59f3-7992-4441-a070-3efb6bf36cb6" (UID: "569b59f3-7992-4441-a070-3efb6bf36cb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:25.129854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.129609    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569b59f3-7992-4441-a070-3efb6bf36cb6-istiod-ca-cert" pod "569b59f3-7992-4441-a070-3efb6bf36cb6" (UID: "569b59f3-7992-4441-a070-3efb6bf36cb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:25.130473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.130450    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-kube-api-access-cc9wm" pod "569b59f3-7992-4441-a070-3efb6bf36cb6" (UID: "569b59f3-7992-4441-a070-3efb6bf36cb6"). InnerVolumeSpecName "kube-api-access-cc9wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:25.224019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.223983    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:25.224019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.224010    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569b59f3-7992-4441-a070-3efb6bf36cb6-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:25.224019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.224019    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:25.224019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.224027    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:25.224321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.224036    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:25.224321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.224044    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:25.224321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.224051    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:25.224321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.224060    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569b59f3-7992-4441-a070-3efb6bf36cb6-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:25.224321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.224068    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc9wm\" (UniqueName: \"kubernetes.io/projected/569b59f3-7992-4441-a070-3efb6bf36cb6-kube-api-access-cc9wm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:25.379691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.379604    4898 generic.go:356] "Generic (PLEG): container finished" podID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerID="8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb" exitCode=0
Nov 02 07:35:25.379691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.379653    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"
Nov 02 07:35:25.379691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.379657    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" event={"ID":"569b59f3-7992-4441-a070-3efb6bf36cb6","Type":"ContainerDied","Data":"8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb"}
Nov 02 07:35:25.379971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.379730    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5" event={"ID":"569b59f3-7992-4441-a070-3efb6bf36cb6","Type":"ContainerDied","Data":"1a2a1b4b4f1f66ffc903a8937eb5a412ce69c2e377597bb13be3f5c27aabb11d"}
Nov 02 07:35:25.379971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.379768    4898 scope.go:122] "RemoveContainer" containerID="8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb"
Nov 02 07:35:25.390704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.390688    4898 scope.go:122] "RemoveContainer" containerID="0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016"
Nov 02 07:35:25.399663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.399644    4898 scope.go:122] "RemoveContainer" containerID="8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb"
Nov 02 07:35:25.399950 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:35:25.399931    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb\": container with ID starting with 8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb not found: ID does not exist" containerID="8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb"
Nov 02 07:35:25.399997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.399958    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb"} err="failed to get container status \"8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb\": rpc error: code = NotFound desc = could not find container \"8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb\": container with ID starting with 8f8f40732b2f0692f9ccecab5210a0eb2cf50e8c8e1e2c57ce78497f6e4a2bbb not found: ID does not exist"
Nov 02 07:35:25.399997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.399974    4898 scope.go:122] "RemoveContainer" containerID="0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016"
Nov 02 07:35:25.400220 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:35:25.400200    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016\": container with ID starting with 0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016 not found: ID does not exist" containerID="0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016"
Nov 02 07:35:25.400256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.400227    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016"} err="failed to get container status \"0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016\": rpc error: code = NotFound desc = could not find container \"0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016\": container with ID starting with 0c962d2f00b3d10874bb81299f0ca9724ac67721b121210a531486e6b9c40016 not found: ID does not exist"
Nov 02 07:35:25.439902 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.439843    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"]
Nov 02 07:35:25.447636 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.447596    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-be1fb7c9-openshift-default-7f46bd7587-dfzl5"]
Nov 02 07:35:25.463160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.463131    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525f3483-1ccf-4e00-8453-e1be717300c3" path="/var/lib/kubelet/pods/525f3483-1ccf-4e00-8453-e1be717300c3/volumes"
Nov 02 07:35:25.463594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:25.463581    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" path="/var/lib/kubelet/pods/569b59f3-7992-4441-a070-3efb6bf36cb6/volumes"
Nov 02 07:35:28.239191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239156    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"]
Nov 02 07:35:28.239616 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239603    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fd3e3dc4-09fe-497e-92a2-21297c58aa1c" containerName="payload-pre-processing"
Nov 02 07:35:28.239660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239617    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3e3dc4-09fe-497e-92a2-21297c58aa1c" containerName="payload-pre-processing"
Nov 02 07:35:28.239660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239641    4898 cpu_manager.go:414] "containerMap: removing container" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy"
Nov 02 07:35:28.239660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239647    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy"
Nov 02 07:35:28.239660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239660    4898 cpu_manager.go:414] "containerMap: removing container" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy"
Nov 02 07:35:28.239786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239666    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy"
Nov 02 07:35:28.239786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239684    4898 cpu_manager.go:414] "containerMap: removing container" podUID="151f52fe-5ecf-417a-8f40-ebffd15ea385" containerName="payload-pre-processing"
Nov 02 07:35:28.239786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239689    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151f52fe-5ecf-417a-8f40-ebffd15ea385" containerName="payload-pre-processing"
Nov 02 07:35:28.239786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239757    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy"
Nov 02 07:35:28.239786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239764    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="569b59f3-7992-4441-a070-3efb6bf36cb6" containerName="istio-proxy"
Nov 02 07:35:28.239786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239773    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="151f52fe-5ecf-417a-8f40-ebffd15ea385" containerName="payload-pre-processing"
Nov 02 07:35:28.239786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.239781    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3e3dc4-09fe-497e-92a2-21297c58aa1c" containerName="payload-pre-processing"
Nov 02 07:35:28.244505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.244487    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"
Nov 02 07:35:28.262352 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.262326    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-9db802\""
Nov 02 07:35:28.275740 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.275532    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"]
Nov 02 07:35:28.290033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.275680    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-9db802-dockercfg-7z6hj\""
Nov 02 07:35:28.318129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.318097    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"]
Nov 02 07:35:28.322180 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.322157    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 07:35:28.343421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.343391    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"]
Nov 02 07:35:28.398485 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.398454    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea92ad5-8888-44e8-b546-a58d39a3d37e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4\" (UID: \"eea92ad5-8888-44e8-b546-a58d39a3d37e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"
Nov 02 07:35:28.398638 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.398513    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q9xh\" (UniqueName: \"kubernetes.io/projected/eea92ad5-8888-44e8-b546-a58d39a3d37e-kube-api-access-2q9xh\") pod \"payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4\" (UID: \"eea92ad5-8888-44e8-b546-a58d39a3d37e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"
Nov 02 07:35:28.500599 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.500513    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9xh\" (UniqueName: \"kubernetes.io/projected/eea92ad5-8888-44e8-b546-a58d39a3d37e-kube-api-access-2q9xh\") pod \"payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4\" (UID: \"eea92ad5-8888-44e8-b546-a58d39a3d37e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"
Nov 02 07:35:28.500763 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.500627    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7kt\" (UniqueName: \"kubernetes.io/projected/d031a556-bb96-4947-bdc9-a96b70bac427-kube-api-access-8c7kt\") pod \"payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99\" (UID: \"d031a556-bb96-4947-bdc9-a96b70bac427\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 07:35:28.500839 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.500759    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d031a556-bb96-4947-bdc9-a96b70bac427-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99\" (UID: \"d031a556-bb96-4947-bdc9-a96b70bac427\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 07:35:28.500839 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.500788    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea92ad5-8888-44e8-b546-a58d39a3d37e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4\" (UID: \"eea92ad5-8888-44e8-b546-a58d39a3d37e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"
Nov 02 07:35:28.502407 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.502378    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea92ad5-8888-44e8-b546-a58d39a3d37e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4\" (UID: \"eea92ad5-8888-44e8-b546-a58d39a3d37e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"
Nov 02 07:35:28.527592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.527280    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9xh\" (UniqueName: \"kubernetes.io/projected/eea92ad5-8888-44e8-b546-a58d39a3d37e-kube-api-access-2q9xh\") pod \"payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4\" (UID: \"eea92ad5-8888-44e8-b546-a58d39a3d37e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"
Nov 02 07:35:28.587415 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.587381    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"
Nov 02 07:35:28.601628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.601603    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7kt\" (UniqueName: \"kubernetes.io/projected/d031a556-bb96-4947-bdc9-a96b70bac427-kube-api-access-8c7kt\") pod \"payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99\" (UID: \"d031a556-bb96-4947-bdc9-a96b70bac427\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 07:35:28.601729 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.601710    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d031a556-bb96-4947-bdc9-a96b70bac427-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99\" (UID: \"d031a556-bb96-4947-bdc9-a96b70bac427\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 07:35:28.602438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.602417    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d031a556-bb96-4947-bdc9-a96b70bac427-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99\" (UID: \"d031a556-bb96-4947-bdc9-a96b70bac427\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 07:35:28.625074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.625043    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7kt\" (UniqueName: \"kubernetes.io/projected/d031a556-bb96-4947-bdc9-a96b70bac427-kube-api-access-8c7kt\") pod \"payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99\" (UID: \"d031a556-bb96-4947-bdc9-a96b70bac427\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 07:35:28.639160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.639126    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 07:35:28.886077 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:35:28.886040    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea92ad5_8888_44e8_b546_a58d39a3d37e.slice/crio-5ff666df3b82f8e8b9b039dfc562f7981f4e0ef1c1dfbadf4e6f413da8ded609 WatchSource:0}: Error finding container 5ff666df3b82f8e8b9b039dfc562f7981f4e0ef1c1dfbadf4e6f413da8ded609: Status 404 returned error can't find the container with id 5ff666df3b82f8e8b9b039dfc562f7981f4e0ef1c1dfbadf4e6f413da8ded609
Nov 02 07:35:28.889073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.887836    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"]
Nov 02 07:35:28.990098 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:28.990062    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"]
Nov 02 07:35:28.994748 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:35:28.994716    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd031a556_bb96_4947_bdc9_a96b70bac427.slice/crio-c3bb54361b7837be0c43befc2d93d013bac1029cd6c3363b0c0145fb5126d2d6 WatchSource:0}: Error finding container c3bb54361b7837be0c43befc2d93d013bac1029cd6c3363b0c0145fb5126d2d6: Status 404 returned error can't find the container with id c3bb54361b7837be0c43befc2d93d013bac1029cd6c3363b0c0145fb5126d2d6
Nov 02 07:35:29.423903 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:29.423850    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99" event={"ID":"d031a556-bb96-4947-bdc9-a96b70bac427","Type":"ContainerStarted","Data":"d7229b4c958a98d898db1c50c8554295b82abbe01b1271e6f6f9f9c523c9bd1b"}
Nov 02 07:35:29.424447 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:29.424068    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 07:35:29.424447 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:29.424107    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99" event={"ID":"d031a556-bb96-4947-bdc9-a96b70bac427","Type":"ContainerStarted","Data":"c3bb54361b7837be0c43befc2d93d013bac1029cd6c3363b0c0145fb5126d2d6"}
Nov 02 07:35:29.426073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:29.426043    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4" event={"ID":"eea92ad5-8888-44e8-b546-a58d39a3d37e","Type":"ContainerStarted","Data":"e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b"}
Nov 02 07:35:29.426205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:29.426103    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"
Nov 02 07:35:29.426205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:29.426124    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4" event={"ID":"eea92ad5-8888-44e8-b546-a58d39a3d37e","Type":"ContainerStarted","Data":"5ff666df3b82f8e8b9b039dfc562f7981f4e0ef1c1dfbadf4e6f413da8ded609"}
Nov 02 07:35:29.486350 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:29.486282    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99" podStartSLOduration=1.486264998 podStartE2EDuration="1.486264998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:29.480405081 +0000 UTC m=+680.170732047" watchObservedRunningTime="2026-11-02 07:35:29.486264998 +0000 UTC m=+680.176591964"
Nov 02 07:35:29.523494 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:29.523447    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4" podStartSLOduration=1.523434343 podStartE2EDuration="1.523434343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:29.521998806 +0000 UTC m=+680.212325770" watchObservedRunningTime="2026-11-02 07:35:29.523434343 +0000 UTC m=+680.213761306"
Nov 02 07:35:30.210043 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.210017    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:35:30.217919 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.217884    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-socket\") pod \"c5445332-4238-4877-a4d1-073fa1cb3bbd\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") "
Nov 02 07:35:30.218020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.217934    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-podinfo\") pod \"c5445332-4238-4877-a4d1-073fa1cb3bbd\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") "
Nov 02 07:35:30.218020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.217990    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-kube-api-access-dk5jn\" (UniqueName: \"kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-kube-api-access-dk5jn\") pod \"c5445332-4238-4877-a4d1-073fa1cb3bbd\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") "
Nov 02 07:35:30.218135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218026    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-envoy\") pod \"c5445332-4238-4877-a4d1-073fa1cb3bbd\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") "
Nov 02 07:35:30.218135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218061    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-data\") pod \"c5445332-4238-4877-a4d1-073fa1cb3bbd\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") "
Nov 02 07:35:30.218135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218091    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-token\" (UniqueName: \"kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-token\") pod \"c5445332-4238-4877-a4d1-073fa1cb3bbd\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") "
Nov 02 07:35:30.218135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218127    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-certs\") pod \"c5445332-4238-4877-a4d1-073fa1cb3bbd\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") "
Nov 02 07:35:30.218347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218143    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-socket" pod "c5445332-4238-4877-a4d1-073fa1cb3bbd" (UID: "c5445332-4238-4877-a4d1-073fa1cb3bbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:30.218347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218181    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5445332-4238-4877-a4d1-073fa1cb3bbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5445332-4238-4877-a4d1-073fa1cb3bbd-istiod-ca-cert\") pod \"c5445332-4238-4877-a4d1-073fa1cb3bbd\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") "
Nov 02 07:35:30.218347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218214    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-credential-socket\") pod \"c5445332-4238-4877-a4d1-073fa1cb3bbd\" (UID: \"c5445332-4238-4877-a4d1-073fa1cb3bbd\") "
Nov 02 07:35:30.218503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218378    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-data" pod "c5445332-4238-4877-a4d1-073fa1cb3bbd" (UID: "c5445332-4238-4877-a4d1-073fa1cb3bbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:30.218503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218443    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-certs" pod "c5445332-4238-4877-a4d1-073fa1cb3bbd" (UID: "c5445332-4238-4877-a4d1-073fa1cb3bbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:30.218601 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218574    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-credential-socket" pod "c5445332-4238-4877-a4d1-073fa1cb3bbd" (UID: "c5445332-4238-4877-a4d1-073fa1cb3bbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:30.218668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218636    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5445332-4238-4877-a4d1-073fa1cb3bbd-istiod-ca-cert" pod "c5445332-4238-4877-a4d1-073fa1cb3bbd" (UID: "c5445332-4238-4877-a4d1-073fa1cb3bbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:35:30.218726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218651    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:30.218726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218701    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:30.218726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.218719    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:30.220428 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.220404    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-envoy" pod "c5445332-4238-4877-a4d1-073fa1cb3bbd" (UID: "c5445332-4238-4877-a4d1-073fa1cb3bbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:35:30.220552 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.220429    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-kube-api-access-dk5jn" pod "c5445332-4238-4877-a4d1-073fa1cb3bbd" (UID: "c5445332-4238-4877-a4d1-073fa1cb3bbd"). InnerVolumeSpecName "kube-api-access-dk5jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:30.220783 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.220759    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-podinfo" pod "c5445332-4238-4877-a4d1-073fa1cb3bbd" (UID: "c5445332-4238-4877-a4d1-073fa1cb3bbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:35:30.220906 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.220785    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-token" pod "c5445332-4238-4877-a4d1-073fa1cb3bbd" (UID: "c5445332-4238-4877-a4d1-073fa1cb3bbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:35:30.320121 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.320079    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk5jn\" (UniqueName: \"kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-kube-api-access-dk5jn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:30.320121 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.320118    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:30.320328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.320133    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:30.320328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.320146    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5445332-4238-4877-a4d1-073fa1cb3bbd-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:30.320328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.320159    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5445332-4238-4877-a4d1-073fa1cb3bbd-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:30.320328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.320180    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5445332-4238-4877-a4d1-073fa1cb3bbd-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:35:30.435612 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.435517    4898 generic.go:356] "Generic (PLEG): container finished" podID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerID="cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580" exitCode=0
Nov 02 07:35:30.436436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.435574    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" event={"ID":"c5445332-4238-4877-a4d1-073fa1cb3bbd","Type":"ContainerDied","Data":"cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580"}
Nov 02 07:35:30.436595 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.436505    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw" event={"ID":"c5445332-4238-4877-a4d1-073fa1cb3bbd","Type":"ContainerDied","Data":"522e296175e324ccbb2d602ff15190e768de1af325d5b4318865ff1f88be98ca"}
Nov 02 07:35:30.436595 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.436541    4898 scope.go:122] "RemoveContainer" containerID="cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580"
Nov 02 07:35:30.436713 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.435611    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"
Nov 02 07:35:30.449628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.449604    4898 scope.go:122] "RemoveContainer" containerID="cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550"
Nov 02 07:35:30.485642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.485620    4898 scope.go:122] "RemoveContainer" containerID="cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580"
Nov 02 07:35:30.486005 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:35:30.485979    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580\": container with ID starting with cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580 not found: ID does not exist" containerID="cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580"
Nov 02 07:35:30.486090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.486014    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580"} err="failed to get container status \"cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580\": rpc error: code = NotFound desc = could not find container \"cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580\": container with ID starting with cecc094d718c92def34d990edccd0931853783f695aafcda2ccda3b56f2f5580 not found: ID does not exist"
Nov 02 07:35:30.486090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.486030    4898 scope.go:122] "RemoveContainer" containerID="cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550"
Nov 02 07:35:30.486352 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:35:30.486333    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550\": container with ID starting with cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550 not found: ID does not exist" containerID="cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550"
Nov 02 07:35:30.486399 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.486352    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550"} err="failed to get container status \"cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550\": rpc error: code = NotFound desc = could not find container \"cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550\": container with ID starting with cd87e628740f0174fc50afe3f83833067b7bddd476f029ede59faad743979550 not found: ID does not exist"
Nov 02 07:35:30.498390 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.498365    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"]
Nov 02 07:35:30.502001 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:30.501977    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8c43c08e-openshift-default-6cd84689b5-7cgsw"]
Nov 02 07:35:31.494773 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.494732    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" path="/var/lib/kubelet/pods/c5445332-4238-4877-a4d1-073fa1cb3bbd/volumes"
Nov 02 07:35:31.539346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.539308    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"]
Nov 02 07:35:31.539918 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.539877    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerName="istio-proxy"
Nov 02 07:35:31.539972 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.539920    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerName="istio-proxy"
Nov 02 07:35:31.539972 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.539931    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerName="istio-proxy"
Nov 02 07:35:31.539972 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.539936    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerName="istio-proxy"
Nov 02 07:35:31.540085 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.540026    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerName="istio-proxy"
Nov 02 07:35:31.540085 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.540035    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5445332-4238-4877-a4d1-073fa1cb3bbd" containerName="istio-proxy"
Nov 02 07:35:31.544754 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.544736    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"
Nov 02 07:35:31.574927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.573460    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-7e7a83\""
Nov 02 07:35:31.586941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.579067    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-7e7a83-dockercfg-n9rxb\""
Nov 02 07:35:31.589014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.588541    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"]
Nov 02 07:35:31.616831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.616804    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"]
Nov 02 07:35:31.621029 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.621006    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"
Nov 02 07:35:31.636774 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.636736    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlwxq\" (UniqueName: \"kubernetes.io/projected/266b1f5a-899e-4175-9358-de87e3f7502a-kube-api-access-xlwxq\") pod \"payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs\" (UID: \"266b1f5a-899e-4175-9358-de87e3f7502a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"
Nov 02 07:35:31.636903 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.636777    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266b1f5a-899e-4175-9358-de87e3f7502a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs\" (UID: \"266b1f5a-899e-4175-9358-de87e3f7502a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"
Nov 02 07:35:31.636983 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.636923    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j58s5\" (UniqueName: \"kubernetes.io/projected/35a88092-f516-49ce-9448-0f5a01ba4174-kube-api-access-j58s5\") pod \"payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd\" (UID: \"35a88092-f516-49ce-9448-0f5a01ba4174\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"
Nov 02 07:35:31.636983 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.636944    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35a88092-f516-49ce-9448-0f5a01ba4174-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd\" (UID: \"35a88092-f516-49ce-9448-0f5a01ba4174\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"
Nov 02 07:35:31.641061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.641037    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"]
Nov 02 07:35:31.738495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.738459    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwxq\" (UniqueName: \"kubernetes.io/projected/266b1f5a-899e-4175-9358-de87e3f7502a-kube-api-access-xlwxq\") pod \"payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs\" (UID: \"266b1f5a-899e-4175-9358-de87e3f7502a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"
Nov 02 07:35:31.738686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.738505    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266b1f5a-899e-4175-9358-de87e3f7502a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs\" (UID: \"266b1f5a-899e-4175-9358-de87e3f7502a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"
Nov 02 07:35:31.738686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.738544    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j58s5\" (UniqueName: \"kubernetes.io/projected/35a88092-f516-49ce-9448-0f5a01ba4174-kube-api-access-j58s5\") pod \"payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd\" (UID: \"35a88092-f516-49ce-9448-0f5a01ba4174\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"
Nov 02 07:35:31.738686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.738563    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35a88092-f516-49ce-9448-0f5a01ba4174-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd\" (UID: \"35a88092-f516-49ce-9448-0f5a01ba4174\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"
Nov 02 07:35:31.739345 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.739319    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35a88092-f516-49ce-9448-0f5a01ba4174-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd\" (UID: \"35a88092-f516-49ce-9448-0f5a01ba4174\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"
Nov 02 07:35:31.739433 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.739326    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266b1f5a-899e-4175-9358-de87e3f7502a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs\" (UID: \"266b1f5a-899e-4175-9358-de87e3f7502a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"
Nov 02 07:35:31.787959 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.786797    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlwxq\" (UniqueName: \"kubernetes.io/projected/266b1f5a-899e-4175-9358-de87e3f7502a-kube-api-access-xlwxq\") pod \"payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs\" (UID: \"266b1f5a-899e-4175-9358-de87e3f7502a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"
Nov 02 07:35:31.787959 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.786834    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j58s5\" (UniqueName: \"kubernetes.io/projected/35a88092-f516-49ce-9448-0f5a01ba4174-kube-api-access-j58s5\") pod \"payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd\" (UID: \"35a88092-f516-49ce-9448-0f5a01ba4174\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"
Nov 02 07:35:31.891934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.891085    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"
Nov 02 07:35:31.940963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:31.936998    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"
Nov 02 07:35:32.283510 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:32.283468    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"]
Nov 02 07:35:32.293124 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:35:32.292820    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a88092_f516_49ce_9448_0f5a01ba4174.slice/crio-f2981b6f0997a7988de93a3b3ac325555c8d5e76711ae780970831d50c6cb86a WatchSource:0}: Error finding container f2981b6f0997a7988de93a3b3ac325555c8d5e76711ae780970831d50c6cb86a: Status 404 returned error can't find the container with id f2981b6f0997a7988de93a3b3ac325555c8d5e76711ae780970831d50c6cb86a
Nov 02 07:35:32.400193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:32.400095    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"]
Nov 02 07:35:32.438942 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:35:32.438706    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266b1f5a_899e_4175_9358_de87e3f7502a.slice/crio-aa58087573376f0049cdf485a3955d6cfda893d3dcc3bf9f0a60035f563bb22c WatchSource:0}: Error finding container aa58087573376f0049cdf485a3955d6cfda893d3dcc3bf9f0a60035f563bb22c: Status 404 returned error can't find the container with id aa58087573376f0049cdf485a3955d6cfda893d3dcc3bf9f0a60035f563bb22c
Nov 02 07:35:32.591793 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:32.591694    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd" event={"ID":"35a88092-f516-49ce-9448-0f5a01ba4174","Type":"ContainerStarted","Data":"f2981b6f0997a7988de93a3b3ac325555c8d5e76711ae780970831d50c6cb86a"}
Nov 02 07:35:32.594014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:32.593944    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs" event={"ID":"266b1f5a-899e-4175-9358-de87e3f7502a","Type":"ContainerStarted","Data":"aa58087573376f0049cdf485a3955d6cfda893d3dcc3bf9f0a60035f563bb22c"}
Nov 02 07:35:33.602135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:33.602091    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd" event={"ID":"35a88092-f516-49ce-9448-0f5a01ba4174","Type":"ContainerStarted","Data":"21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37"}
Nov 02 07:35:33.602630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:33.602163    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"
Nov 02 07:35:33.603772 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:33.603744    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs" event={"ID":"266b1f5a-899e-4175-9358-de87e3f7502a","Type":"ContainerStarted","Data":"f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584"}
Nov 02 07:35:33.603901 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:33.603792    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"
Nov 02 07:35:33.647630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:33.647570    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd" podStartSLOduration=2.64755512 podStartE2EDuration="2.64755512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:33.643803466 +0000 UTC m=+684.334130431" watchObservedRunningTime="2026-11-02 07:35:33.64755512 +0000 UTC m=+684.337882085"
Nov 02 07:35:34.393371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:34.392114    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"
Nov 02 07:35:34.393371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:34.392743    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"
Nov 02 07:35:34.434957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:34.434879    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs" podStartSLOduration=3.43486553 podStartE2EDuration="3.43486553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:33.690687968 +0000 UTC m=+684.381014933" watchObservedRunningTime="2026-11-02 07:35:34.43486553 +0000 UTC m=+685.125192485"
Nov 02 07:35:37.635605 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:37.635513    4898 generic.go:356] "Generic (PLEG): container finished" podID="53de2d77-31bc-415c-8c29-d768438375af" containerID="dc11653148133e1dbbce433fdadc2fca8b60c9be24672213d9897bc89c53322d" exitCode=0
Nov 02 07:35:37.636034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:37.635603    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" event={"ID":"53de2d77-31bc-415c-8c29-d768438375af","Type":"ContainerDied","Data":"dc11653148133e1dbbce433fdadc2fca8b60c9be24672213d9897bc89c53322d"}
Nov 02 07:35:37.636146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:37.636126    4898 scope.go:122] "RemoveContainer" containerID="dc11653148133e1dbbce433fdadc2fca8b60c9be24672213d9897bc89c53322d"
Nov 02 07:35:37.637702 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:37.637660    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:37.637789 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:37.637771    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:37.637850 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:37.637834    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:38.646134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:38.646091    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" event={"ID":"53de2d77-31bc-415c-8c29-d768438375af","Type":"ContainerStarted","Data":"812e1dab2a4d367460238b96357e1ec9af08e4f88bbb93298a73c4157bff655b"}
Nov 02 07:35:38.916997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:38.916953    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:38.917214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:38.917079    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 07:35:38.917214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:38.917121    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" podUID="53de2d77-31bc-415c-8c29-d768438375af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 07:35:39.917176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:39.917133    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 07:35:39.917581 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:39.917190    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" podUID="53de2d77-31bc-415c-8c29-d768438375af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 07:35:40.439908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:40.439750    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 07:35:40.439908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:40.439822    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"
Nov 02 07:35:40.917362 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:40.917330    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 07:35:40.917739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:40.917379    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" podUID="53de2d77-31bc-415c-8c29-d768438375af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 07:35:40.981913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:40.977841    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"]
Nov 02 07:35:40.993385 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:40.993349    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"]
Nov 02 07:35:40.993556 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:40.993471    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.010611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.010581    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"openshift-service-ca.crt\""
Nov 02 07:35:41.010762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.010677    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"simulator-w3-354660-kserve-self-signed-certs\""
Nov 02 07:35:41.010762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.010740    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"kube-root-ca.crt\""
Nov 02 07:35:41.022328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.022298    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"default-dockercfg-kkdsp\""
Nov 02 07:35:41.117403 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.117364    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9j2\" (UniqueName: \"kubernetes.io/projected/cd11c253-5683-46aa-9b84-40ad7d77303b-kube-api-access-gr9j2\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.117571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.117422    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd11c253-5683-46aa-9b84-40ad7d77303b-tls-certs\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.117618 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.117564    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-dshm\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.117681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.117666    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-model-cache\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.117720 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.117713    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-home\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.117763 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.117735    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-tmp-dir\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.218585 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.218494    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-model-cache\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.218585 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.218552    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-home\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.218585 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.218573    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-tmp-dir\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.218805 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.218597    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9j2\" (UniqueName: \"kubernetes.io/projected/cd11c253-5683-46aa-9b84-40ad7d77303b-kube-api-access-gr9j2\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.218805 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.218624    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd11c253-5683-46aa-9b84-40ad7d77303b-tls-certs\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.218805 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.218657    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-dshm\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.219479 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.219436    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-tmp-dir\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.219582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.219543    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-model-cache\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.219861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.219835    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-home\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.222068 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.221768    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd11c253-5683-46aa-9b84-40ad7d77303b-tls-certs\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.222418 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.222395    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-dshm\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.242727 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.242700    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9j2\" (UniqueName: \"kubernetes.io/projected/cd11c253-5683-46aa-9b84-40ad7d77303b-kube-api-access-gr9j2\") pod \"simulator-w3-354660-kserve-5f5cb69cdb-2xlgr\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") " pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.315016 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.314965    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:41.645044 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:35:41.645014    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd11c253_5683_46aa_9b84_40ad7d77303b.slice/crio-4e3a3363e7812264fa2c68ffed651a831a70c8d961ebfb5ef271aa0a330dc601 WatchSource:0}: Error finding container 4e3a3363e7812264fa2c68ffed651a831a70c8d961ebfb5ef271aa0a330dc601: Status 404 returned error can't find the container with id 4e3a3363e7812264fa2c68ffed651a831a70c8d961ebfb5ef271aa0a330dc601
Nov 02 07:35:41.645534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.645513    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"]
Nov 02 07:35:41.702237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.702207    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" event={"ID":"cd11c253-5683-46aa-9b84-40ad7d77303b","Type":"ContainerStarted","Data":"4e3a3363e7812264fa2c68ffed651a831a70c8d961ebfb5ef271aa0a330dc601"}
Nov 02 07:35:41.916602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.916569    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 07:35:41.916753 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:41.916626    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" podUID="53de2d77-31bc-415c-8c29-d768438375af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 07:35:42.720342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:42.720298    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" event={"ID":"cd11c253-5683-46aa-9b84-40ad7d77303b","Type":"ContainerStarted","Data":"52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735"}
Nov 02 07:35:42.775913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:42.775088    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podStartSLOduration=2.775069133 podStartE2EDuration="2.775069133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:42.772117685 +0000 UTC m=+693.462444650" watchObservedRunningTime="2026-11-02 07:35:42.775069133 +0000 UTC m=+693.465396097"
Nov 02 07:35:42.916814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:42.916776    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 02 07:35:42.917003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:42.916833    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" podUID="53de2d77-31bc-415c-8c29-d768438375af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 02 07:35:43.316177 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:43.316132    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:43.727639 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:43.727603    4898 generic.go:356] "Generic (PLEG): container finished" podID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerID="dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d" exitCode=0
Nov 02 07:35:43.728210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:43.727674    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" event={"ID":"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e","Type":"ContainerDied","Data":"dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d"}
Nov 02 07:35:43.728210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:43.728132    4898 scope.go:122] "RemoveContainer" containerID="dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d"
Nov 02 07:35:43.729604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:43.729562    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:43.729702 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:43.729663    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:43.729767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:43.729715    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:35:43.921287 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:43.921256    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:43.921659 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:43.921637    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:43.922241 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:43.922223    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:35:44.612915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:44.612866    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"
Nov 02 07:35:44.613116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:44.613082    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"
Nov 02 07:35:44.738751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:44.738711    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" event={"ID":"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e","Type":"ContainerStarted","Data":"7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c"}
Nov 02 07:35:44.914392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:44.914348    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:44.915944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:44.915912    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 02 07:35:44.916102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:44.915971    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 02 07:35:45.004679 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:35:45.004638    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:35:45.914651 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:45.914609    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 02 07:35:45.915090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:45.914677    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 02 07:35:46.914690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:46.914643    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 02 07:35:46.915238 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:46.914697    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 02 07:35:47.315530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:47.315443    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:47.914954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:47.914919    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 02 07:35:47.915365 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:47.914995    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 02 07:35:48.804723 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.804690    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"]
Nov 02 07:35:48.809668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.809645    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.828884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.828853    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-model-cache\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.829029 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.828923    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-home\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.829029 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.828952    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-dshm\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.829029 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.829010    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tmp-dir\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.829260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.829035    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rthnm\" (UniqueName: \"kubernetes.io/projected/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-kube-api-access-rthnm\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.829260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.829066    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tls-certs\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.830037 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.830007    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"]
Nov 02 07:35:48.839788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.839763    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0fbc17\"/\"simulator-w0-0fbc17-kserve-self-signed-certs\""
Nov 02 07:35:48.839954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.839850    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0fbc17\"/\"kube-root-ca.crt\""
Nov 02 07:35:48.839954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.839868    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0fbc17\"/\"default-dockercfg-k8rx2\""
Nov 02 07:35:48.839954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.839867    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0fbc17\"/\"openshift-service-ca.crt\""
Nov 02 07:35:48.915073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.915045    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 02 07:35:48.915430 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.915095    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 02 07:35:48.929803 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.929773    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-home\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.929964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.929809    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-dshm\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.929964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.929858    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tmp-dir\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.929964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.929882    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rthnm\" (UniqueName: \"kubernetes.io/projected/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-kube-api-access-rthnm\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.929964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.929925    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tls-certs\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.930196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.930005    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-model-cache\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.930257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.930195    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-home\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.930314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.930264    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tmp-dir\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.930360 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.930333    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-model-cache\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.932219 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.932192    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-dshm\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.932389 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.932373    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tls-certs\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:48.953786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:48.953761    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rthnm\" (UniqueName: \"kubernetes.io/projected/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-kube-api-access-rthnm\") pod \"simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") " pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:49.123610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:49.123519    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:49.447024 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:35:49.446992    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34cee5d4_7fa9_4857_9916_49ab2a89f8e0.slice/crio-b3d5a357644dee547326dbba1f4fd4513686e93820902f615c57719d99761c15 WatchSource:0}: Error finding container b3d5a357644dee547326dbba1f4fd4513686e93820902f615c57719d99761c15: Status 404 returned error can't find the container with id b3d5a357644dee547326dbba1f4fd4513686e93820902f615c57719d99761c15
Nov 02 07:35:49.447721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:49.447695    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"]
Nov 02 07:35:49.794184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:49.794096    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" event={"ID":"34cee5d4-7fa9-4857-9916-49ab2a89f8e0","Type":"ContainerStarted","Data":"c6e808c347ec39d11c99a28a11c1f6b1a2fc93cfb1c3a01778e4ef253a5e12b2"}
Nov 02 07:35:49.794184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:49.794143    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" event={"ID":"34cee5d4-7fa9-4857-9916-49ab2a89f8e0","Type":"ContainerStarted","Data":"b3d5a357644dee547326dbba1f4fd4513686e93820902f615c57719d99761c15"}
Nov 02 07:35:49.838205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:49.838153    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podStartSLOduration=1.838138742 podStartE2EDuration="1.838138742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:49.836144326 +0000 UTC m=+700.526471291" watchObservedRunningTime="2026-11-02 07:35:49.838138742 +0000 UTC m=+700.528465743"
Nov 02 07:35:49.920207 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:49.920164    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:49.921386 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:49.921361    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 07:35:50.124075 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.123992    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:50.562196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.562157    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"]
Nov 02 07:35:50.566243 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.566212    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.580644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.580612    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"]
Nov 02 07:35:50.589477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.589445    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-337764\"/\"openshift-service-ca.crt\""
Nov 02 07:35:50.589633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.589557    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-337764\"/\"kube-root-ca.crt\""
Nov 02 07:35:50.593966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.593716    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-337764\"/\"default-dockercfg-5lt4c\""
Nov 02 07:35:50.593966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.593767    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-337764\"/\"simulator-w2-337764-kserve-self-signed-certs\""
Nov 02 07:35:50.653873 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.653834    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tmp-dir\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.653873 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.653869    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-dshm\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.654193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.653960    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tls-certs\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.654193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.654034    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn847\" (UniqueName: \"kubernetes.io/projected/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-kube-api-access-xn847\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.654193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.654102    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-model-cache\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.654353 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.654232    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-home\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.754827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.754789    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tmp-dir\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.754827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.754829    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-dshm\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.755103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.754854    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tls-certs\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.755103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.754870    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn847\" (UniqueName: \"kubernetes.io/projected/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-kube-api-access-xn847\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.755103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.755046    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-model-cache\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.755289 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.755269    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-home\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.755537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.755507    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tmp-dir\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.755664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.755556    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-model-cache\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.755730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.755663    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-home\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.757260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.757237    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-dshm\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.757615 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.757594    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tls-certs\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.779780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.779678    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn847\" (UniqueName: \"kubernetes.io/projected/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-kube-api-access-xn847\") pod \"simulator-w2-337764-kserve-64d7c65d99-t6fvb\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") " pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:50.882036 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:50.881952    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:51.191704 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:35:51.191677    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c5688fd_298b_49d4_b852_4c4f6d4f40d4.slice/crio-5abaf14488d2eac556d03792e7c8a206e9a3a006db7bb2990eeabae87916efbe WatchSource:0}: Error finding container 5abaf14488d2eac556d03792e7c8a206e9a3a006db7bb2990eeabae87916efbe: Status 404 returned error can't find the container with id 5abaf14488d2eac556d03792e7c8a206e9a3a006db7bb2990eeabae87916efbe
Nov 02 07:35:51.193232 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:51.193204    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"]
Nov 02 07:35:51.315327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:51.315295    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:51.328841 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:51.328817    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:51.822594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:51.822549    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" event={"ID":"9c5688fd-298b-49d4-b852-4c4f6d4f40d4","Type":"ContainerStarted","Data":"cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788"}
Nov 02 07:35:51.822594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:51.822598    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" event={"ID":"9c5688fd-298b-49d4-b852-4c4f6d4f40d4","Type":"ContainerStarted","Data":"5abaf14488d2eac556d03792e7c8a206e9a3a006db7bb2990eeabae87916efbe"}
Nov 02 07:35:51.833752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:51.833722    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:35:51.865093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:51.865046    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podStartSLOduration=1.865033767 podStartE2EDuration="1.865033767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:35:51.863701529 +0000 UTC m=+702.554028512" watchObservedRunningTime="2026-11-02 07:35:51.865033767 +0000 UTC m=+702.555360731"
Nov 02 07:35:51.883017 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:51.882999    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:35:59.124244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:59.124208    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:59.137686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:59.137658    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:35:59.898904 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:35:59.898871    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:36:00.882901 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:00.882848    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:36:00.896777 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:00.896748    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:36:00.908592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:00.908563    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:36:14.989308 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:36:14.989254    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:36:16.081147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.081111    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"]
Nov 02 07:36:16.087385 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.087360    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.106715 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.106681    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"]
Nov 02 07:36:16.115269 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.115243    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0fbc17\"/\"premium-w0-0fbc17-kserve-self-signed-certs\""
Nov 02 07:36:16.168708 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.168675    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tmp-dir\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.168909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.168738    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-model-cache\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.168909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.168844    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tls-certs\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.169024 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.168996    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-dshm\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.169150 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.169069    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtz9h\" (UniqueName: \"kubernetes.io/projected/11ddd010-79e1-4281-b3ba-a5c0f273fde1-kube-api-access-vtz9h\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.169199 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.169163    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-home\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.269823 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.269785    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tmp-dir\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.270033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.269847    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-model-cache\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.270033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.269929    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tls-certs\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.270033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.270011    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-dshm\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.271163 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.270035    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtz9h\" (UniqueName: \"kubernetes.io/projected/11ddd010-79e1-4281-b3ba-a5c0f273fde1-kube-api-access-vtz9h\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.271163 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.270093    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-home\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.271163 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.270436    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-model-cache\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.271163 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.270514    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-home\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.271163 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.270662    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tmp-dir\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.287949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.282163    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-dshm\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.289871 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.289808    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tls-certs\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.299952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.299316    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtz9h\" (UniqueName: \"kubernetes.io/projected/11ddd010-79e1-4281-b3ba-a5c0f273fde1-kube-api-access-vtz9h\") pod \"premium-w0-0fbc17-kserve-58c44cd85-nrtk7\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") " pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.401170 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.401141    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:16.701636 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:36:16.701606    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11ddd010_79e1_4281_b3ba_a5c0f273fde1.slice/crio-f12368db957966a8429a19c18b338938ac9ea720e735436b6d4b035e7466986c WatchSource:0}: Error finding container f12368db957966a8429a19c18b338938ac9ea720e735436b6d4b035e7466986c: Status 404 returned error can't find the container with id f12368db957966a8429a19c18b338938ac9ea720e735436b6d4b035e7466986c
Nov 02 07:36:16.702954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:16.702927    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"]
Nov 02 07:36:17.026693 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:17.026609    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" event={"ID":"11ddd010-79e1-4281-b3ba-a5c0f273fde1","Type":"ContainerStarted","Data":"429e49c5d6d57d3297caf95da943cc459a1018f4bba61946ce759d109a49a468"}
Nov 02 07:36:17.026693 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:17.026650    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" event={"ID":"11ddd010-79e1-4281-b3ba-a5c0f273fde1","Type":"ContainerStarted","Data":"f12368db957966a8429a19c18b338938ac9ea720e735436b6d4b035e7466986c"}
Nov 02 07:36:17.069541 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:17.069493    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podStartSLOduration=1.069479209 podStartE2EDuration="1.069479209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:36:17.067601693 +0000 UTC m=+727.757928656" watchObservedRunningTime="2026-11-02 07:36:17.069479209 +0000 UTC m=+727.759806174"
Nov 02 07:36:17.401694 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:17.401663    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:26.401794 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:26.401754    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:26.414792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:26.414762    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:27.112717 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:27.112684    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:36:44.992142 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:36:44.992092    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:36:52.990433 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:52.990352    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"]
Nov 02 07:36:52.995521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:52.995502    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:53.014865 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.014835    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"]
Nov 02 07:36:53.023394 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.023365    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:36:53.023544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.023470    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:36:53.023544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.023517    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:36:53.023665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.023597    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:36:53.104686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.104645    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08592b61-ba78-4240-8436-91ac9458863c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f377b3a2-h255n\" (UID: \"08592b61-ba78-4240-8436-91ac9458863c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:53.104854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.104717    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08592b61-ba78-4240-8436-91ac9458863c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f377b3a2-h255n\" (UID: \"08592b61-ba78-4240-8436-91ac9458863c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:53.104854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.104808    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt5f5\" (UniqueName: \"kubernetes.io/projected/08592b61-ba78-4240-8436-91ac9458863c-kube-api-access-rt5f5\") pod \"maas-api-revoke-sub-f377b3a2-h255n\" (UID: \"08592b61-ba78-4240-8436-91ac9458863c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:53.205857 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.205826    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08592b61-ba78-4240-8436-91ac9458863c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f377b3a2-h255n\" (UID: \"08592b61-ba78-4240-8436-91ac9458863c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:53.206031 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.205924    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08592b61-ba78-4240-8436-91ac9458863c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f377b3a2-h255n\" (UID: \"08592b61-ba78-4240-8436-91ac9458863c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:53.206031 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.205992    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt5f5\" (UniqueName: \"kubernetes.io/projected/08592b61-ba78-4240-8436-91ac9458863c-kube-api-access-rt5f5\") pod \"maas-api-revoke-sub-f377b3a2-h255n\" (UID: \"08592b61-ba78-4240-8436-91ac9458863c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:53.206624 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.206595    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08592b61-ba78-4240-8436-91ac9458863c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f377b3a2-h255n\" (UID: \"08592b61-ba78-4240-8436-91ac9458863c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:53.208197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.208162    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08592b61-ba78-4240-8436-91ac9458863c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f377b3a2-h255n\" (UID: \"08592b61-ba78-4240-8436-91ac9458863c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:53.229242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.229218    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt5f5\" (UniqueName: \"kubernetes.io/projected/08592b61-ba78-4240-8436-91ac9458863c-kube-api-access-rt5f5\") pod \"maas-api-revoke-sub-f377b3a2-h255n\" (UID: \"08592b61-ba78-4240-8436-91ac9458863c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:53.309682 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.309605    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:53.608698 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:36:53.608670    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08592b61_ba78_4240_8436_91ac9458863c.slice/crio-2f7ac50c98133799e251ca5850f11ab64919e60486365d0973ededa0ff4fea61 WatchSource:0}: Error finding container 2f7ac50c98133799e251ca5850f11ab64919e60486365d0973ededa0ff4fea61: Status 404 returned error can't find the container with id 2f7ac50c98133799e251ca5850f11ab64919e60486365d0973ededa0ff4fea61
Nov 02 07:36:53.609391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:53.609364    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"]
Nov 02 07:36:54.303200 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:54.303155    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n" event={"ID":"08592b61-ba78-4240-8436-91ac9458863c","Type":"ContainerStarted","Data":"c47e40b8d2523bcd35eb68524f434f228f79f63d66959def2c4548d1f7f1978b"}
Nov 02 07:36:54.303670 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:54.303240    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n" event={"ID":"08592b61-ba78-4240-8436-91ac9458863c","Type":"ContainerStarted","Data":"2f7ac50c98133799e251ca5850f11ab64919e60486365d0973ededa0ff4fea61"}
Nov 02 07:36:54.345361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:54.345311    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n" podStartSLOduration=2.345298554 podStartE2EDuration="2.345298554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:36:54.343821325 +0000 UTC m=+765.034148288" watchObservedRunningTime="2026-11-02 07:36:54.345298554 +0000 UTC m=+765.035625517"
Nov 02 07:36:55.311723 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:55.311686    4898 generic.go:356] "Generic (PLEG): container finished" podID="08592b61-ba78-4240-8436-91ac9458863c" containerID="c47e40b8d2523bcd35eb68524f434f228f79f63d66959def2c4548d1f7f1978b" exitCode=0
Nov 02 07:36:55.312156 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:55.311764    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n" event={"ID":"08592b61-ba78-4240-8436-91ac9458863c","Type":"ContainerDied","Data":"c47e40b8d2523bcd35eb68524f434f228f79f63d66959def2c4548d1f7f1978b"}
Nov 02 07:36:56.463568 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:56.463543    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:56.541794 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:56.541759    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08592b61-ba78-4240-8436-91ac9458863c-kube-api-access-rt5f5\" (UniqueName: \"kubernetes.io/projected/08592b61-ba78-4240-8436-91ac9458863c-kube-api-access-rt5f5\") pod \"08592b61-ba78-4240-8436-91ac9458863c\" (UID: \"08592b61-ba78-4240-8436-91ac9458863c\") "
Nov 02 07:36:56.542015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:56.541931    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08592b61-ba78-4240-8436-91ac9458863c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08592b61-ba78-4240-8436-91ac9458863c-maas-api-xks-ca\") pod \"08592b61-ba78-4240-8436-91ac9458863c\" (UID: \"08592b61-ba78-4240-8436-91ac9458863c\") "
Nov 02 07:36:56.542015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:56.542006    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08592b61-ba78-4240-8436-91ac9458863c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08592b61-ba78-4240-8436-91ac9458863c-maas-api-openshift-ca\") pod \"08592b61-ba78-4240-8436-91ac9458863c\" (UID: \"08592b61-ba78-4240-8436-91ac9458863c\") "
Nov 02 07:36:56.542412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:56.542377    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08592b61-ba78-4240-8436-91ac9458863c-maas-api-openshift-ca" pod "08592b61-ba78-4240-8436-91ac9458863c" (UID: "08592b61-ba78-4240-8436-91ac9458863c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:36:56.544124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:56.544095    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08592b61-ba78-4240-8436-91ac9458863c-maas-api-xks-ca" pod "08592b61-ba78-4240-8436-91ac9458863c" (UID: "08592b61-ba78-4240-8436-91ac9458863c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:36:56.544124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:56.544108    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08592b61-ba78-4240-8436-91ac9458863c-kube-api-access-rt5f5" pod "08592b61-ba78-4240-8436-91ac9458863c" (UID: "08592b61-ba78-4240-8436-91ac9458863c"). InnerVolumeSpecName "kube-api-access-rt5f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:36:56.643751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:56.643679    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt5f5\" (UniqueName: \"kubernetes.io/projected/08592b61-ba78-4240-8436-91ac9458863c-kube-api-access-rt5f5\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:36:56.643751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:56.643704    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08592b61-ba78-4240-8436-91ac9458863c-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:36:56.643751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:56.643723    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08592b61-ba78-4240-8436-91ac9458863c-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:36:57.326575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:57.326545    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"
Nov 02 07:36:57.326740 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:57.326588    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n" event={"ID":"08592b61-ba78-4240-8436-91ac9458863c","Type":"ContainerDied","Data":"2f7ac50c98133799e251ca5850f11ab64919e60486365d0973ededa0ff4fea61"}
Nov 02 07:36:57.326740 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:36:57.326644    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f7ac50c98133799e251ca5850f11ab64919e60486365d0973ededa0ff4fea61"
Nov 02 07:37:03.639081 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.639044    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"]
Nov 02 07:37:03.639593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.639574    4898 cpu_manager.go:414] "containerMap: removing container" podUID="08592b61-ba78-4240-8436-91ac9458863c" containerName="revoke-keys"
Nov 02 07:37:03.639637 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.639595    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08592b61-ba78-4240-8436-91ac9458863c" containerName="revoke-keys"
Nov 02 07:37:03.639721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.639711    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="08592b61-ba78-4240-8436-91ac9458863c" containerName="revoke-keys"
Nov 02 07:37:03.643114 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.643097    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.659622 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.659600    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-337764-openshift-default-dockercfg-45fm2\""
Nov 02 07:37:03.674902 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.674863    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"]
Nov 02 07:37:03.743513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.743478    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.743663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.743525    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534095d2-847d-4712-8f8d-acb19d774364-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.743663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.743550    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534095d2-847d-4712-8f8d-acb19d774364-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.743742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.743682    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.743742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.743725    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.743813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.743792    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.743863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.743850    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj4wz\" (UniqueName: \"kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-kube-api-access-hj4wz\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.743936 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.743909    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.743987 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.743962    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.845373 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.845341    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.845553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.845390    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534095d2-847d-4712-8f8d-acb19d774364-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.845553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.845428    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534095d2-847d-4712-8f8d-acb19d774364-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.845553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.845477    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.845553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.845522    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.845974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.845558    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.845974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.845600    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj4wz\" (UniqueName: \"kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-kube-api-access-hj4wz\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.845974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.845657    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.845974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.845690    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.845974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.845806    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.846326 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.846212    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.846326 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.846263    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534095d2-847d-4712-8f8d-acb19d774364-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.846399 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.846332    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.846459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.846435    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.848020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.847999    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.848259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.848241    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534095d2-847d-4712-8f8d-acb19d774364-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.869200 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.869179    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.869393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.869373    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj4wz\" (UniqueName: \"kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-kube-api-access-hj4wz\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:03.956482 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:03.956453    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:04.272784 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:04.272752    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"]
Nov 02 07:37:04.274116 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:37:04.274087    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534095d2_847d_4712_8f8d_acb19d774364.slice/crio-49fa92c15594fc7fa722a3f15e209a9f087f23d355c295534941155d615c17be WatchSource:0}: Error finding container 49fa92c15594fc7fa722a3f15e209a9f087f23d355c295534941155d615c17be: Status 404 returned error can't find the container with id 49fa92c15594fc7fa722a3f15e209a9f087f23d355c295534941155d615c17be
Nov 02 07:37:04.276475 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:04.276436    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:37:04.276572 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:04.276505    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:37:04.276572 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:04.276533    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:37:04.403348 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:04.403309    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" event={"ID":"534095d2-847d-4712-8f8d-acb19d774364","Type":"ContainerStarted","Data":"03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f"}
Nov 02 07:37:04.403532 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:04.403388    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" event={"ID":"534095d2-847d-4712-8f8d-acb19d774364","Type":"ContainerStarted","Data":"49fa92c15594fc7fa722a3f15e209a9f087f23d355c295534941155d615c17be"}
Nov 02 07:37:04.452283 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:04.452222    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" podStartSLOduration=1.452205656 podStartE2EDuration="1.452205656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:04.448849047 +0000 UTC m=+775.139176032" watchObservedRunningTime="2026-11-02 07:37:04.452205656 +0000 UTC m=+775.142532614"
Nov 02 07:37:04.962726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:04.958976    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:05.962915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:05.960861    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:37:05.962915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:05.960960    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" podUID="534095d2-847d-4712-8f8d-acb19d774364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 07:37:06.959920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:06.959203    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:37:06.959920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:06.959263    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" podUID="534095d2-847d-4712-8f8d-acb19d774364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 07:37:07.961253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:07.961218    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:37:07.961833 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:07.961797    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" podUID="534095d2-847d-4712-8f8d-acb19d774364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 07:37:08.253148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.253071    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"]
Nov 02 07:37:08.264363 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.264335    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:08.284814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.284779    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"]
Nov 02 07:37:08.286711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.286689    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:37:08.286993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.286976    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:37:08.287198 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.287184    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:37:08.299097 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.299077    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:37:08.422538 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.422492    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj8vs\" (UniqueName: \"kubernetes.io/projected/fb098e81-4393-4754-a778-2a8234a3c035-kube-api-access-hj8vs\") pod \"maas-api-revoke-sub-ada4a680-gf8g7\" (UID: \"fb098e81-4393-4754-a778-2a8234a3c035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:08.422787 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.422625    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada4a680-gf8g7\" (UID: \"fb098e81-4393-4754-a778-2a8234a3c035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:08.422787 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.422745    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada4a680-gf8g7\" (UID: \"fb098e81-4393-4754-a778-2a8234a3c035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:08.524203 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.524114    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj8vs\" (UniqueName: \"kubernetes.io/projected/fb098e81-4393-4754-a778-2a8234a3c035-kube-api-access-hj8vs\") pod \"maas-api-revoke-sub-ada4a680-gf8g7\" (UID: \"fb098e81-4393-4754-a778-2a8234a3c035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:08.524203 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.524167    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada4a680-gf8g7\" (UID: \"fb098e81-4393-4754-a778-2a8234a3c035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:08.524407 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.524224    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada4a680-gf8g7\" (UID: \"fb098e81-4393-4754-a778-2a8234a3c035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:08.524817 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.524791    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada4a680-gf8g7\" (UID: \"fb098e81-4393-4754-a778-2a8234a3c035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:08.526559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.526537    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada4a680-gf8g7\" (UID: \"fb098e81-4393-4754-a778-2a8234a3c035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:08.547221 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.547198    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj8vs\" (UniqueName: \"kubernetes.io/projected/fb098e81-4393-4754-a778-2a8234a3c035-kube-api-access-hj8vs\") pod \"maas-api-revoke-sub-ada4a680-gf8g7\" (UID: \"fb098e81-4393-4754-a778-2a8234a3c035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:08.607617 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.607576    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:08.943550 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.943523    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"]
Nov 02 07:37:08.973921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.973297    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:37:08.973921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:08.973370    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" podUID="534095d2-847d-4712-8f8d-acb19d774364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 07:37:09.441959 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:09.441924    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7" event={"ID":"fb098e81-4393-4754-a778-2a8234a3c035","Type":"ContainerStarted","Data":"4c7e2ad0a9b80c08b98daa90b00a3b63ccc5726aaaf0a71976f403ea574c2ef6"}
Nov 02 07:37:09.442141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:09.441986    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7" event={"ID":"fb098e81-4393-4754-a778-2a8234a3c035","Type":"ContainerStarted","Data":"195b614db928fbb93a2dbcaa8d8e1a59b873ce38750b6383ff8c17747572d152"}
Nov 02 07:37:09.487686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:09.487630    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7" podStartSLOduration=1.487613156 podStartE2EDuration="1.487613156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:09.486324304 +0000 UTC m=+780.176651269" watchObservedRunningTime="2026-11-02 07:37:09.487613156 +0000 UTC m=+780.177940121"
Nov 02 07:37:09.964961 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:09.964728    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 07:37:09.964961 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:09.964785    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" podUID="534095d2-847d-4712-8f8d-acb19d774364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 07:37:10.458567 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:10.458527    4898 generic.go:356] "Generic (PLEG): container finished" podID="fb098e81-4393-4754-a778-2a8234a3c035" containerID="4c7e2ad0a9b80c08b98daa90b00a3b63ccc5726aaaf0a71976f403ea574c2ef6" exitCode=0
Nov 02 07:37:10.459135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:10.458623    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7" event={"ID":"fb098e81-4393-4754-a778-2a8234a3c035","Type":"ContainerDied","Data":"4c7e2ad0a9b80c08b98daa90b00a3b63ccc5726aaaf0a71976f403ea574c2ef6"}
Nov 02 07:37:11.046381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.046344    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:11.046856 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.046830    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:11.048135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.048116    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:37:11.504337 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.504304    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"]
Nov 02 07:37:11.508354 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.508331    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.523127 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.523104    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7e7a83-openshift-default-dockercfg-8bmgw\""
Nov 02 07:37:11.543933 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.532883    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"]
Nov 02 07:37:11.570983 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.570947    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.571138 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.571030    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.571138 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.571084    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrm26\" (UniqueName: \"kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-kube-api-access-jrm26\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.571138 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.571132    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.571328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.571158    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42283595-1739-4f03-845c-41f9ef5f5073-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.571328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.571200    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42283595-1739-4f03-845c-41f9ef5f5073-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.571328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.571228    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.571328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.571286    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.571328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.571321    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.623391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.623368    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:11.671956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.671925    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb098e81-4393-4754-a778-2a8234a3c035-kube-api-access-hj8vs\" (UniqueName: \"kubernetes.io/projected/fb098e81-4393-4754-a778-2a8234a3c035-kube-api-access-hj8vs\") pod \"fb098e81-4393-4754-a778-2a8234a3c035\" (UID: \"fb098e81-4393-4754-a778-2a8234a3c035\") "
Nov 02 07:37:11.672098 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672042    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-openshift-ca\") pod \"fb098e81-4393-4754-a778-2a8234a3c035\" (UID: \"fb098e81-4393-4754-a778-2a8234a3c035\") "
Nov 02 07:37:11.672098 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672084    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-xks-ca\") pod \"fb098e81-4393-4754-a778-2a8234a3c035\" (UID: \"fb098e81-4393-4754-a778-2a8234a3c035\") "
Nov 02 07:37:11.672277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672250    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42283595-1739-4f03-845c-41f9ef5f5073-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.672357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672308    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.672414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672377    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.672472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672418    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.672472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672440    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-openshift-ca" pod "fb098e81-4393-4754-a778-2a8234a3c035" (UID: "fb098e81-4393-4754-a778-2a8234a3c035"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:37:11.672572 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672532    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.672624 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672611    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.672779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672753    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrm26\" (UniqueName: \"kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-kube-api-access-jrm26\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.672916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672817    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.672916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672841    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42283595-1739-4f03-845c-41f9ef5f5073-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.673086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.672959    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:11.673086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.673009    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.673296 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.673274    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.673551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.673527    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.674233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.674202    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.675194 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.675170    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.675602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.675580    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-xks-ca" pod "fb098e81-4393-4754-a778-2a8234a3c035" (UID: "fb098e81-4393-4754-a778-2a8234a3c035"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:37:11.675980 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.675957    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42283595-1739-4f03-845c-41f9ef5f5073-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.677927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.677903    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb098e81-4393-4754-a778-2a8234a3c035-kube-api-access-hj8vs" pod "fb098e81-4393-4754-a778-2a8234a3c035" (UID: "fb098e81-4393-4754-a778-2a8234a3c035"). InnerVolumeSpecName "kube-api-access-hj8vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:37:11.686618 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.686592    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42283595-1739-4f03-845c-41f9ef5f5073-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.696362 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.696339    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.696618 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.696599    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrm26\" (UniqueName: \"kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-kube-api-access-jrm26\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:11.774226 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.774151    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb098e81-4393-4754-a778-2a8234a3c035-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:11.774226 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.774187    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj8vs\" (UniqueName: \"kubernetes.io/projected/fb098e81-4393-4754-a778-2a8234a3c035-kube-api-access-hj8vs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:11.822466 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:11.822436    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:12.141377 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:37:12.141343    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42283595_1739_4f03_845c_41f9ef5f5073.slice/crio-8bf25ed73fb8b37214e203277baa4f2348e2e53a3c650266d6f5a0b1b4a5e764 WatchSource:0}: Error finding container 8bf25ed73fb8b37214e203277baa4f2348e2e53a3c650266d6f5a0b1b4a5e764: Status 404 returned error can't find the container with id 8bf25ed73fb8b37214e203277baa4f2348e2e53a3c650266d6f5a0b1b4a5e764
Nov 02 07:37:12.142655 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:12.142631    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"]
Nov 02 07:37:12.143803 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:12.143772    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:37:12.143875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:12.143837    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:37:12.143875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:12.143868    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:37:12.475032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:12.474981    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" event={"ID":"42283595-1739-4f03-845c-41f9ef5f5073","Type":"ContainerStarted","Data":"bf78c593adae2369c199bdc0b6a968ed6128774808343cc3a379bd382e34c17a"}
Nov 02 07:37:12.475243 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:12.475061    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" event={"ID":"42283595-1739-4f03-845c-41f9ef5f5073","Type":"ContainerStarted","Data":"8bf25ed73fb8b37214e203277baa4f2348e2e53a3c650266d6f5a0b1b4a5e764"}
Nov 02 07:37:12.477050 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:12.477017    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"
Nov 02 07:37:12.477194 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:12.477080    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7" event={"ID":"fb098e81-4393-4754-a778-2a8234a3c035","Type":"ContainerDied","Data":"195b614db928fbb93a2dbcaa8d8e1a59b873ce38750b6383ff8c17747572d152"}
Nov 02 07:37:12.477194 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:12.477129    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="195b614db928fbb93a2dbcaa8d8e1a59b873ce38750b6383ff8c17747572d152"
Nov 02 07:37:12.520866 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:12.520819    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" podStartSLOduration=1.520805836 podStartE2EDuration="1.520805836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:12.519506985 +0000 UTC m=+783.209833951" watchObservedRunningTime="2026-11-02 07:37:12.520805836 +0000 UTC m=+783.211132859"
Nov 02 07:37:12.822837 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:12.822757    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:13.825110 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:13.825074    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 07:37:13.825501 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:13.825139    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" podUID="42283595-1739-4f03-845c-41f9ef5f5073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 07:37:14.823678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:14.823643    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 07:37:14.824507 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:14.823951    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" podUID="42283595-1739-4f03-845c-41f9ef5f5073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 07:37:15.027010 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:37:15.026543    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:37:15.823076 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:15.823038    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 07:37:15.823377 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:15.823331    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" podUID="42283595-1739-4f03-845c-41f9ef5f5073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 07:37:16.823681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:16.823641    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 07:37:16.824153 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:16.823706    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" podUID="42283595-1739-4f03-845c-41f9ef5f5073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 07:37:17.823218 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:17.823170    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 07:37:17.823422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:17.823233    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" podUID="42283595-1739-4f03-845c-41f9ef5f5073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 07:37:18.753038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.753000    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"]
Nov 02 07:37:18.753578 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.753559    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fb098e81-4393-4754-a778-2a8234a3c035" containerName="revoke-keys"
Nov 02 07:37:18.753578 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.753579    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb098e81-4393-4754-a778-2a8234a3c035" containerName="revoke-keys"
Nov 02 07:37:18.753705 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.753695    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb098e81-4393-4754-a778-2a8234a3c035" containerName="revoke-keys"
Nov 02 07:37:18.757999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.757981    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:18.774929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.774878    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:37:18.779259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.779231    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"]
Nov 02 07:37:18.785543 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.785512    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:37:18.823003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.822973    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 02 07:37:18.823136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.823034    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" podUID="42283595-1739-4f03-845c-41f9ef5f5073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 02 07:37:18.827766 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.827741    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvlng\" (UniqueName: \"kubernetes.io/projected/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-kube-api-access-wvlng\") pod \"maas-api-revoke-sub-50829d69-55xvj\" (UID: \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:18.827914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.827786    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50829d69-55xvj\" (UID: \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:18.827964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.827946    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50829d69-55xvj\" (UID: \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:18.847219 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.847194    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:37:18.847310 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.847249    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:37:18.929077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.929045    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvlng\" (UniqueName: \"kubernetes.io/projected/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-kube-api-access-wvlng\") pod \"maas-api-revoke-sub-50829d69-55xvj\" (UID: \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:18.929262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.929096    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50829d69-55xvj\" (UID: \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:18.929262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.929166    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50829d69-55xvj\" (UID: \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:18.929953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.929934    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50829d69-55xvj\" (UID: \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:18.931637 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.931616    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50829d69-55xvj\" (UID: \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:18.954850 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:18.954824    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvlng\" (UniqueName: \"kubernetes.io/projected/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-kube-api-access-wvlng\") pod \"maas-api-revoke-sub-50829d69-55xvj\" (UID: \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:19.082558 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:19.082464    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:19.396843 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:37:19.396807    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26c7f4f_7a4b_4d82_acc7_6335c1f221d9.slice/crio-c3727f4017f442722a3e80d8329096ad2b894a8ca17309363116b6cc8eae086c WatchSource:0}: Error finding container c3727f4017f442722a3e80d8329096ad2b894a8ca17309363116b6cc8eae086c: Status 404 returned error can't find the container with id c3727f4017f442722a3e80d8329096ad2b894a8ca17309363116b6cc8eae086c
Nov 02 07:37:19.397830 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:19.397803    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"]
Nov 02 07:37:19.560745 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:19.560707    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj" event={"ID":"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9","Type":"ContainerStarted","Data":"c3727f4017f442722a3e80d8329096ad2b894a8ca17309363116b6cc8eae086c"}
Nov 02 07:37:19.828239 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:19.828210    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:19.828721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:19.828558    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:19.829918 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:19.829867    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:37:20.571975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:20.571942    4898 generic.go:356] "Generic (PLEG): container finished" podID="e26c7f4f-7a4b-4d82-acc7-6335c1f221d9" containerID="0cd584d44984a09034f254c2e3cd99963042c00b0335cd00331475bd655c18a8" exitCode=0
Nov 02 07:37:20.572157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:20.572031    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj" event={"ID":"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9","Type":"ContainerDied","Data":"0cd584d44984a09034f254c2e3cd99963042c00b0335cd00331475bd655c18a8"}
Nov 02 07:37:21.242691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.242656    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"]
Nov 02 07:37:21.246960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.246939    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.264411 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.264381    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9db802-openshift-default-dockercfg-4p9sf\""
Nov 02 07:37:21.280619 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.280589    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"]
Nov 02 07:37:21.363611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.363575    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.363793 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.363619    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.363793 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.363655    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.363793 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.363731    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9p9d\" (UniqueName: \"kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-kube-api-access-r9p9d\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.363793 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.363773    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.364000 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.363884    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.364000 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.363928    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.364069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.364048    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.364103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.364086    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21181cd6-c08a-4db8-b18b-8ca7b0743867-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.465226 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.465185    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.465226 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.465231    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.465478 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.465273    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.465478 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.465294    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21181cd6-c08a-4db8-b18b-8ca7b0743867-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.465609 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.465584    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.465702 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.465669    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.465830 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.465775    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.465830 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.465805    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.466006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.465849    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.466006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.465880    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9p9d\" (UniqueName: \"kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-kube-api-access-r9p9d\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.466006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.465934    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.466173 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.466048    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.466173 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.466158    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21181cd6-c08a-4db8-b18b-8ca7b0743867-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.466324 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.466305    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.468222 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.468192    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.468318 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.468238    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.490872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.490847    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.494160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.494092    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9p9d\" (UniqueName: \"kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-kube-api-access-r9p9d\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.560613 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.560584    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:21.567389 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.567361    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"]
Nov 02 07:37:21.571342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.571321    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:21.601347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.600266    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"]
Nov 02 07:37:21.669712 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.667454    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c85fd61a-d87rj\" (UID: \"3a40a65b-52bc-425c-903f-410b8f0c784a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:21.669712 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.667522    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c85fd61a-d87rj\" (UID: \"3a40a65b-52bc-425c-903f-410b8f0c784a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:21.669712 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.667785    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4lb\" (UniqueName: \"kubernetes.io/projected/3a40a65b-52bc-425c-903f-410b8f0c784a-kube-api-access-zp4lb\") pod \"maas-api-revoke-sub-c85fd61a-d87rj\" (UID: \"3a40a65b-52bc-425c-903f-410b8f0c784a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:21.768941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.768828    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4lb\" (UniqueName: \"kubernetes.io/projected/3a40a65b-52bc-425c-903f-410b8f0c784a-kube-api-access-zp4lb\") pod \"maas-api-revoke-sub-c85fd61a-d87rj\" (UID: \"3a40a65b-52bc-425c-903f-410b8f0c784a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:21.769130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.768978    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c85fd61a-d87rj\" (UID: \"3a40a65b-52bc-425c-903f-410b8f0c784a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:21.769130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.769010    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c85fd61a-d87rj\" (UID: \"3a40a65b-52bc-425c-903f-410b8f0c784a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:21.769795 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.769770    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c85fd61a-d87rj\" (UID: \"3a40a65b-52bc-425c-903f-410b8f0c784a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:21.771466 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.771443    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c85fd61a-d87rj\" (UID: \"3a40a65b-52bc-425c-903f-410b8f0c784a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:21.793063 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.793032    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4lb\" (UniqueName: \"kubernetes.io/projected/3a40a65b-52bc-425c-903f-410b8f0c784a-kube-api-access-zp4lb\") pod \"maas-api-revoke-sub-c85fd61a-d87rj\" (UID: \"3a40a65b-52bc-425c-903f-410b8f0c784a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:21.794971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.794948    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:21.869820 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.869783    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-xks-ca\") pod \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\" (UID: \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\") "
Nov 02 07:37:21.870039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.869981    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-kube-api-access-wvlng\" (UniqueName: \"kubernetes.io/projected/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-kube-api-access-wvlng\") pod \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\" (UID: \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\") "
Nov 02 07:37:21.870570 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.870532    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-openshift-ca\") pod \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\" (UID: \"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9\") "
Nov 02 07:37:21.870922 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.870861    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-openshift-ca" pod "e26c7f4f-7a4b-4d82-acc7-6335c1f221d9" (UID: "e26c7f4f-7a4b-4d82-acc7-6335c1f221d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:37:21.871472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.871449    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:21.872505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.872480    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-xks-ca" pod "e26c7f4f-7a4b-4d82-acc7-6335c1f221d9" (UID: "e26c7f4f-7a4b-4d82-acc7-6335c1f221d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:37:21.872721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.872691    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-kube-api-access-wvlng" pod "e26c7f4f-7a4b-4d82-acc7-6335c1f221d9" (UID: "e26c7f4f-7a4b-4d82-acc7-6335c1f221d9"). InnerVolumeSpecName "kube-api-access-wvlng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:37:21.896285 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.896261    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"]
Nov 02 07:37:21.896956 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:37:21.896931    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21181cd6_c08a_4db8_b18b_8ca7b0743867.slice/crio-c727c01d7396ecba722cbb1a39b867ca338f58596ae02468931d794c58f54629 WatchSource:0}: Error finding container c727c01d7396ecba722cbb1a39b867ca338f58596ae02468931d794c58f54629: Status 404 returned error can't find the container with id c727c01d7396ecba722cbb1a39b867ca338f58596ae02468931d794c58f54629
Nov 02 07:37:21.900221 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.900175    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:37:21.900348 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.900265    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:37:21.900348 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.900312    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:37:21.931719 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.931694    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:21.972673 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.972638    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:21.972673 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:21.972677    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvlng\" (UniqueName: \"kubernetes.io/projected/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9-kube-api-access-wvlng\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:22.237037 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:37:22.237006    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a40a65b_52bc_425c_903f_410b8f0c784a.slice/crio-9426f90a812098d92c8f7b871bbb07ac9e56929fce1549d01f59f3a0ec73bcf3 WatchSource:0}: Error finding container 9426f90a812098d92c8f7b871bbb07ac9e56929fce1549d01f59f3a0ec73bcf3: Status 404 returned error can't find the container with id 9426f90a812098d92c8f7b871bbb07ac9e56929fce1549d01f59f3a0ec73bcf3
Nov 02 07:37:22.237688 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:22.237664    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"]
Nov 02 07:37:22.591269 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:22.591235    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj" event={"ID":"3a40a65b-52bc-425c-903f-410b8f0c784a","Type":"ContainerStarted","Data":"9426f90a812098d92c8f7b871bbb07ac9e56929fce1549d01f59f3a0ec73bcf3"}
Nov 02 07:37:22.592997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:22.592963    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" event={"ID":"21181cd6-c08a-4db8-b18b-8ca7b0743867","Type":"ContainerStarted","Data":"c2e753dcb2eaa01ad783ca25bed39a9e7602026a508bfd8a949f9fd5060de936"}
Nov 02 07:37:22.593143 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:22.593014    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" event={"ID":"21181cd6-c08a-4db8-b18b-8ca7b0743867","Type":"ContainerStarted","Data":"c727c01d7396ecba722cbb1a39b867ca338f58596ae02468931d794c58f54629"}
Nov 02 07:37:22.594312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:22.594291    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj" event={"ID":"e26c7f4f-7a4b-4d82-acc7-6335c1f221d9","Type":"ContainerDied","Data":"c3727f4017f442722a3e80d8329096ad2b894a8ca17309363116b6cc8eae086c"}
Nov 02 07:37:22.594409 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:22.594312    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"
Nov 02 07:37:22.594409 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:22.594323    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3727f4017f442722a3e80d8329096ad2b894a8ca17309363116b6cc8eae086c"
Nov 02 07:37:22.640259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:22.640194    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" podStartSLOduration=1.640175799 podStartE2EDuration="1.640175799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:22.637252568 +0000 UTC m=+793.327579534" watchObservedRunningTime="2026-11-02 07:37:22.640175799 +0000 UTC m=+793.330502763"
Nov 02 07:37:23.573970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:23.564276    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:23.573970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:23.568769    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:37:23.573970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:23.568820    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 07:37:23.605836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:23.605801    4898 generic.go:356] "Generic (PLEG): container finished" podID="3a40a65b-52bc-425c-903f-410b8f0c784a" containerID="ecfa67f733a75e6b602965e4fdd437d2c04ddce1c6b4d617886d77c905b79122" exitCode=0
Nov 02 07:37:23.606224 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:23.605880    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj" event={"ID":"3a40a65b-52bc-425c-903f-410b8f0c784a","Type":"ContainerDied","Data":"ecfa67f733a75e6b602965e4fdd437d2c04ddce1c6b4d617886d77c905b79122"}
Nov 02 07:37:24.561242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:24.561205    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:37:24.561546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:24.561514    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 07:37:24.828366 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:24.828299    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:24.927643 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:24.927615    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-xks-ca\") pod \"3a40a65b-52bc-425c-903f-410b8f0c784a\" (UID: \"3a40a65b-52bc-425c-903f-410b8f0c784a\") "
Nov 02 07:37:24.927814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:24.927703    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a40a65b-52bc-425c-903f-410b8f0c784a-kube-api-access-zp4lb\" (UniqueName: \"kubernetes.io/projected/3a40a65b-52bc-425c-903f-410b8f0c784a-kube-api-access-zp4lb\") pod \"3a40a65b-52bc-425c-903f-410b8f0c784a\" (UID: \"3a40a65b-52bc-425c-903f-410b8f0c784a\") "
Nov 02 07:37:24.927814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:24.927800    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-openshift-ca\") pod \"3a40a65b-52bc-425c-903f-410b8f0c784a\" (UID: \"3a40a65b-52bc-425c-903f-410b8f0c784a\") "
Nov 02 07:37:24.928163 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:24.928139    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-openshift-ca" pod "3a40a65b-52bc-425c-903f-410b8f0c784a" (UID: "3a40a65b-52bc-425c-903f-410b8f0c784a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:37:24.929761 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:24.929736    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-xks-ca" pod "3a40a65b-52bc-425c-903f-410b8f0c784a" (UID: "3a40a65b-52bc-425c-903f-410b8f0c784a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:37:24.930099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:24.929836    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a40a65b-52bc-425c-903f-410b8f0c784a-kube-api-access-zp4lb" pod "3a40a65b-52bc-425c-903f-410b8f0c784a" (UID: "3a40a65b-52bc-425c-903f-410b8f0c784a"). InnerVolumeSpecName "kube-api-access-zp4lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:37:25.029179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:25.029144    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:25.029179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:25.029173    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp4lb\" (UniqueName: \"kubernetes.io/projected/3a40a65b-52bc-425c-903f-410b8f0c784a-kube-api-access-zp4lb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:25.029179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:25.029185    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a40a65b-52bc-425c-903f-410b8f0c784a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:25.564275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:25.564238    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:37:25.564540 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:25.564507    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 07:37:25.621718 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:25.621682    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"
Nov 02 07:37:25.621873 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:25.621725    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj" event={"ID":"3a40a65b-52bc-425c-903f-410b8f0c784a","Type":"ContainerDied","Data":"9426f90a812098d92c8f7b871bbb07ac9e56929fce1549d01f59f3a0ec73bcf3"}
Nov 02 07:37:25.621873 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:25.621772    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9426f90a812098d92c8f7b871bbb07ac9e56929fce1549d01f59f3a0ec73bcf3"
Nov 02 07:37:26.563305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:26.563253    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:37:26.563938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:26.563878    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 07:37:27.577912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:27.571280    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 07:37:27.577912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:27.571337    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 07:37:29.229503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:29.229470    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:29.229934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:29.229911    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:29.230793 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:29.230770    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 07:37:36.289912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.287112    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"]
Nov 02 07:37:36.289912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.287938    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3a40a65b-52bc-425c-903f-410b8f0c784a" containerName="revoke-keys"
Nov 02 07:37:36.289912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.287959    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a40a65b-52bc-425c-903f-410b8f0c784a" containerName="revoke-keys"
Nov 02 07:37:36.289912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.287986    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e26c7f4f-7a4b-4d82-acc7-6335c1f221d9" containerName="revoke-keys"
Nov 02 07:37:36.289912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.287994    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26c7f4f-7a4b-4d82-acc7-6335c1f221d9" containerName="revoke-keys"
Nov 02 07:37:36.289912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.288147    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a40a65b-52bc-425c-903f-410b8f0c784a" containerName="revoke-keys"
Nov 02 07:37:36.289912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.288164    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26c7f4f-7a4b-4d82-acc7-6335c1f221d9" containerName="revoke-keys"
Nov 02 07:37:36.293466 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.293444    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:36.314160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.314127    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"]
Nov 02 07:37:36.316116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.316094    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:37:36.317042 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.317023    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:37:36.317342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.317324    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:37:36.317529 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.317512    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:37:36.393635 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.393599    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edbb1d83-vmsqs\" (UID: \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:36.400909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.400246    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edbb1d83-vmsqs\" (UID: \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:36.400909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.400339    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrcp2\" (UniqueName: \"kubernetes.io/projected/ddcf02e0-410d-4ac7-8214-50da19a83fbb-kube-api-access-nrcp2\") pod \"maas-api-revoke-sub-edbb1d83-vmsqs\" (UID: \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:36.512917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.509023    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrcp2\" (UniqueName: \"kubernetes.io/projected/ddcf02e0-410d-4ac7-8214-50da19a83fbb-kube-api-access-nrcp2\") pod \"maas-api-revoke-sub-edbb1d83-vmsqs\" (UID: \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:36.512917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.509202    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edbb1d83-vmsqs\" (UID: \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:36.512917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.509255    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edbb1d83-vmsqs\" (UID: \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:36.524912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.524627    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edbb1d83-vmsqs\" (UID: \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:36.525147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.525121    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edbb1d83-vmsqs\" (UID: \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:36.554671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.554586    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrcp2\" (UniqueName: \"kubernetes.io/projected/ddcf02e0-410d-4ac7-8214-50da19a83fbb-kube-api-access-nrcp2\") pod \"maas-api-revoke-sub-edbb1d83-vmsqs\" (UID: \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:36.629664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.629625    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:36.954351 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:37:36.954320    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddcf02e0_410d_4ac7_8214_50da19a83fbb.slice/crio-bae4d3b699aaf5191b01c6b143a049baa255baaa73a3a373bf0b30932f37cfea WatchSource:0}: Error finding container bae4d3b699aaf5191b01c6b143a049baa255baaa73a3a373bf0b30932f37cfea: Status 404 returned error can't find the container with id bae4d3b699aaf5191b01c6b143a049baa255baaa73a3a373bf0b30932f37cfea
Nov 02 07:37:36.955120 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:36.955097    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"]
Nov 02 07:37:37.750852 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:37.750811    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs" event={"ID":"ddcf02e0-410d-4ac7-8214-50da19a83fbb","Type":"ContainerStarted","Data":"35a43a480d3311c55e544b4f96ae79491237d5769787c618644717404bfc4273"}
Nov 02 07:37:37.751312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:37.750870    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs" event={"ID":"ddcf02e0-410d-4ac7-8214-50da19a83fbb","Type":"ContainerStarted","Data":"bae4d3b699aaf5191b01c6b143a049baa255baaa73a3a373bf0b30932f37cfea"}
Nov 02 07:37:37.792699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:37.792653    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs" podStartSLOduration=1.792637552 podStartE2EDuration="1.792637552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:37.79156803 +0000 UTC m=+808.481895014" watchObservedRunningTime="2026-11-02 07:37:37.792637552 +0000 UTC m=+808.482964558"
Nov 02 07:37:38.759868 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:38.759825    4898 generic.go:356] "Generic (PLEG): container finished" podID="ddcf02e0-410d-4ac7-8214-50da19a83fbb" containerID="35a43a480d3311c55e544b4f96ae79491237d5769787c618644717404bfc4273" exitCode=0
Nov 02 07:37:38.760290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:38.759923    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs" event={"ID":"ddcf02e0-410d-4ac7-8214-50da19a83fbb","Type":"ContainerDied","Data":"35a43a480d3311c55e544b4f96ae79491237d5769787c618644717404bfc4273"}
Nov 02 07:37:39.315720 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.315686    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"]
Nov 02 07:37:39.319799 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.319779    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:39.339684 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.339658    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"]
Nov 02 07:37:39.460874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.460842    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba23cf6d-dk6vl\" (UID: \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:39.461041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.460883    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba23cf6d-dk6vl\" (UID: \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:39.461129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.461039    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rfm7\" (UniqueName: \"kubernetes.io/projected/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-kube-api-access-6rfm7\") pod \"maas-api-revoke-sub-ba23cf6d-dk6vl\" (UID: \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:39.562534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.562457    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rfm7\" (UniqueName: \"kubernetes.io/projected/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-kube-api-access-6rfm7\") pod \"maas-api-revoke-sub-ba23cf6d-dk6vl\" (UID: \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:39.562713 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.562629    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba23cf6d-dk6vl\" (UID: \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:39.562713 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.562667    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba23cf6d-dk6vl\" (UID: \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:39.563513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.563484    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba23cf6d-dk6vl\" (UID: \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:39.565814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.565752    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba23cf6d-dk6vl\" (UID: \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:39.585905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.585868    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rfm7\" (UniqueName: \"kubernetes.io/projected/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-kube-api-access-6rfm7\") pod \"maas-api-revoke-sub-ba23cf6d-dk6vl\" (UID: \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:39.637075 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.637048    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:39.945703 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:37:39.945673    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbbd712_ab89_4aee_8d0b_fb15e56e6c00.slice/crio-72a2a4bda090b9d88bc337e0268b37c3984cf9aa3bacd54b034c3be94411d53e WatchSource:0}: Error finding container 72a2a4bda090b9d88bc337e0268b37c3984cf9aa3bacd54b034c3be94411d53e: Status 404 returned error can't find the container with id 72a2a4bda090b9d88bc337e0268b37c3984cf9aa3bacd54b034c3be94411d53e
Nov 02 07:37:39.946387 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.946363    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"]
Nov 02 07:37:39.959977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:39.959953    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:40.071293 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.071260    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-openshift-ca\") pod \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\" (UID: \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\") "
Nov 02 07:37:40.071475 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.071426    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-xks-ca\") pod \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\" (UID: \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\") "
Nov 02 07:37:40.071647 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.071626    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddcf02e0-410d-4ac7-8214-50da19a83fbb-kube-api-access-nrcp2\" (UniqueName: \"kubernetes.io/projected/ddcf02e0-410d-4ac7-8214-50da19a83fbb-kube-api-access-nrcp2\") pod \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\" (UID: \"ddcf02e0-410d-4ac7-8214-50da19a83fbb\") "
Nov 02 07:37:40.071727 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.071659    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-openshift-ca" pod "ddcf02e0-410d-4ac7-8214-50da19a83fbb" (UID: "ddcf02e0-410d-4ac7-8214-50da19a83fbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:37:40.072093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.072069    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:40.074059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.074029    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-xks-ca" pod "ddcf02e0-410d-4ac7-8214-50da19a83fbb" (UID: "ddcf02e0-410d-4ac7-8214-50da19a83fbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:37:40.074159 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.074067    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddcf02e0-410d-4ac7-8214-50da19a83fbb-kube-api-access-nrcp2" pod "ddcf02e0-410d-4ac7-8214-50da19a83fbb" (UID: "ddcf02e0-410d-4ac7-8214-50da19a83fbb"). InnerVolumeSpecName "kube-api-access-nrcp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:37:40.173488 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.173458    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddcf02e0-410d-4ac7-8214-50da19a83fbb-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:40.173588 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.173492    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrcp2\" (UniqueName: \"kubernetes.io/projected/ddcf02e0-410d-4ac7-8214-50da19a83fbb-kube-api-access-nrcp2\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:40.784259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.784217    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs" event={"ID":"ddcf02e0-410d-4ac7-8214-50da19a83fbb","Type":"ContainerDied","Data":"bae4d3b699aaf5191b01c6b143a049baa255baaa73a3a373bf0b30932f37cfea"}
Nov 02 07:37:40.784453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.784269    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"
Nov 02 07:37:40.784453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.784278    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bae4d3b699aaf5191b01c6b143a049baa255baaa73a3a373bf0b30932f37cfea"
Nov 02 07:37:40.786103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.786075    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl" event={"ID":"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00","Type":"ContainerStarted","Data":"b93fd397c59656a24e5b9c12658fd4e02872f6eec4704b70bb9c61295d3e4818"}
Nov 02 07:37:40.786241 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.786123    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl" event={"ID":"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00","Type":"ContainerStarted","Data":"72a2a4bda090b9d88bc337e0268b37c3984cf9aa3bacd54b034c3be94411d53e"}
Nov 02 07:37:40.828452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:40.828364    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl" podStartSLOduration=1.82835143 podStartE2EDuration="1.82835143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:40.826027262 +0000 UTC m=+811.516354245" watchObservedRunningTime="2026-11-02 07:37:40.82835143 +0000 UTC m=+811.518678393"
Nov 02 07:37:41.795587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:41.795556    4898 generic.go:356] "Generic (PLEG): container finished" podID="7dbbd712-ab89-4aee-8d0b-fb15e56e6c00" containerID="b93fd397c59656a24e5b9c12658fd4e02872f6eec4704b70bb9c61295d3e4818" exitCode=0
Nov 02 07:37:41.796091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:41.795614    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl" event={"ID":"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00","Type":"ContainerDied","Data":"b93fd397c59656a24e5b9c12658fd4e02872f6eec4704b70bb9c61295d3e4818"}
Nov 02 07:37:43.014787 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.014750    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:43.046158 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.046130    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-openshift-ca\") pod \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\" (UID: \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\") "
Nov 02 07:37:43.046327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.046171    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-xks-ca\") pod \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\" (UID: \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\") "
Nov 02 07:37:43.046327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.046258    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-kube-api-access-6rfm7\" (UniqueName: \"kubernetes.io/projected/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-kube-api-access-6rfm7\") pod \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\" (UID: \"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00\") "
Nov 02 07:37:43.046508 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.046483    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-openshift-ca" pod "7dbbd712-ab89-4aee-8d0b-fb15e56e6c00" (UID: "7dbbd712-ab89-4aee-8d0b-fb15e56e6c00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:37:43.046711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.046661    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:43.048401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.048378    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-xks-ca" pod "7dbbd712-ab89-4aee-8d0b-fb15e56e6c00" (UID: "7dbbd712-ab89-4aee-8d0b-fb15e56e6c00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:37:43.048513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.048469    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-kube-api-access-6rfm7" pod "7dbbd712-ab89-4aee-8d0b-fb15e56e6c00" (UID: "7dbbd712-ab89-4aee-8d0b-fb15e56e6c00"). InnerVolumeSpecName "kube-api-access-6rfm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:37:43.147496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.147459    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:43.147665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.147500    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rfm7\" (UniqueName: \"kubernetes.io/projected/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00-kube-api-access-6rfm7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:43.824587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.824492    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl" event={"ID":"7dbbd712-ab89-4aee-8d0b-fb15e56e6c00","Type":"ContainerDied","Data":"72a2a4bda090b9d88bc337e0268b37c3984cf9aa3bacd54b034c3be94411d53e"}
Nov 02 07:37:43.824834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.824815    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72a2a4bda090b9d88bc337e0268b37c3984cf9aa3bacd54b034c3be94411d53e"
Nov 02 07:37:43.825023 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:43.825008    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"
Nov 02 07:37:45.005067 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:37:45.004980    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:37:49.625592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.625556    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"]
Nov 02 07:37:49.626238 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.626115    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7dbbd712-ab89-4aee-8d0b-fb15e56e6c00" containerName="revoke-keys"
Nov 02 07:37:49.626238 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.626129    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dbbd712-ab89-4aee-8d0b-fb15e56e6c00" containerName="revoke-keys"
Nov 02 07:37:49.626238 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.626153    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ddcf02e0-410d-4ac7-8214-50da19a83fbb" containerName="revoke-keys"
Nov 02 07:37:49.626238 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.626158    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddcf02e0-410d-4ac7-8214-50da19a83fbb" containerName="revoke-keys"
Nov 02 07:37:49.626455 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.626247    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddcf02e0-410d-4ac7-8214-50da19a83fbb" containerName="revoke-keys"
Nov 02 07:37:49.626455 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.626260    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dbbd712-ab89-4aee-8d0b-fb15e56e6c00" containerName="revoke-keys"
Nov 02 07:37:49.629965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.629935    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.659066 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.659032    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-337764\"/\"e2e-conflict-a-d0b940c5-kserve-self-signed-certs\""
Nov 02 07:37:49.664094 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.664066    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"]
Nov 02 07:37:49.768792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.768761    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgdcb\" (UniqueName: \"kubernetes.io/projected/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-kube-api-access-tgdcb\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.768999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.768804    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-home\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.769080 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.769057    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-model-cache\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.769142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.769121    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-dshm\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.769257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.769241    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tls-certs\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.769342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.769288    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tmp-dir\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.805846 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.805817    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"]
Nov 02 07:37:49.810188 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.810169    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:49.836517 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.836486    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:37:49.839164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.839136    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:37:49.839498 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.839479    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:37:49.839682 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.839665    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:37:49.846671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.846555    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"]
Nov 02 07:37:49.872314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.870770    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-model-cache\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.872314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.870840    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-dshm\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.872314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.870871    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tls-certs\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.872314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.870929    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tmp-dir\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.872314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.871012    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgdcb\" (UniqueName: \"kubernetes.io/projected/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-kube-api-access-tgdcb\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.872314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.871039    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-home\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.872314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.871633    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-home\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.872314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.871911    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-model-cache\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.874028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.874004    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tmp-dir\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.899988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.898065    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgdcb\" (UniqueName: \"kubernetes.io/projected/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-kube-api-access-tgdcb\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.899988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.898357    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-dshm\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.906358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.905422    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tls-certs\") pod \"e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") " pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.943536 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.943507    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:49.972416 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.972386    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17be9830-fcwsh\" (UID: \"b2f5d62b-8341-483f-8f30-1c96b01001c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:49.972546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.972460    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msnt7\" (UniqueName: \"kubernetes.io/projected/b2f5d62b-8341-483f-8f30-1c96b01001c8-kube-api-access-msnt7\") pod \"maas-api-revoke-sub-17be9830-fcwsh\" (UID: \"b2f5d62b-8341-483f-8f30-1c96b01001c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:49.972546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:49.972489    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17be9830-fcwsh\" (UID: \"b2f5d62b-8341-483f-8f30-1c96b01001c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:50.073699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.073653    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17be9830-fcwsh\" (UID: \"b2f5d62b-8341-483f-8f30-1c96b01001c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:50.073873 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.073774    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msnt7\" (UniqueName: \"kubernetes.io/projected/b2f5d62b-8341-483f-8f30-1c96b01001c8-kube-api-access-msnt7\") pod \"maas-api-revoke-sub-17be9830-fcwsh\" (UID: \"b2f5d62b-8341-483f-8f30-1c96b01001c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:50.073873 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.073814    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17be9830-fcwsh\" (UID: \"b2f5d62b-8341-483f-8f30-1c96b01001c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:50.074575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.074543    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17be9830-fcwsh\" (UID: \"b2f5d62b-8341-483f-8f30-1c96b01001c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:50.076225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.076198    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17be9830-fcwsh\" (UID: \"b2f5d62b-8341-483f-8f30-1c96b01001c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:50.097806 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.097777    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msnt7\" (UniqueName: \"kubernetes.io/projected/b2f5d62b-8341-483f-8f30-1c96b01001c8-kube-api-access-msnt7\") pod \"maas-api-revoke-sub-17be9830-fcwsh\" (UID: \"b2f5d62b-8341-483f-8f30-1c96b01001c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:50.131178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.131133    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:50.252262 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:37:50.252225    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae0eae37_8d3a_48da_ab8f_6b7bee43f176.slice/crio-14a00f25407c70871e324ba83f965aece8c9ba8d15257167dc684e9f39f50445 WatchSource:0}: Error finding container 14a00f25407c70871e324ba83f965aece8c9ba8d15257167dc684e9f39f50445: Status 404 returned error can't find the container with id 14a00f25407c70871e324ba83f965aece8c9ba8d15257167dc684e9f39f50445
Nov 02 07:37:50.254707 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.254682    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"]
Nov 02 07:37:50.439484 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.439449    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"]
Nov 02 07:37:50.443717 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:37:50.443675    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f5d62b_8341_483f_8f30_1c96b01001c8.slice/crio-f37789667cd46340ec1842f3d4f121205477f1b9342551753f0f5c4af01f00f4 WatchSource:0}: Error finding container f37789667cd46340ec1842f3d4f121205477f1b9342551753f0f5c4af01f00f4: Status 404 returned error can't find the container with id f37789667cd46340ec1842f3d4f121205477f1b9342551753f0f5c4af01f00f4
Nov 02 07:37:50.949950 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.949864    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj" event={"ID":"ae0eae37-8d3a-48da-ab8f-6b7bee43f176","Type":"ContainerStarted","Data":"931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a"}
Nov 02 07:37:50.950436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.949967    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj" event={"ID":"ae0eae37-8d3a-48da-ab8f-6b7bee43f176","Type":"ContainerStarted","Data":"14a00f25407c70871e324ba83f965aece8c9ba8d15257167dc684e9f39f50445"}
Nov 02 07:37:50.952021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.951990    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh" event={"ID":"b2f5d62b-8341-483f-8f30-1c96b01001c8","Type":"ContainerStarted","Data":"745455174c9b1f34d264fbdc85ac3f9f33ff4f929f4905d745d9dc76bb57c1e0"}
Nov 02 07:37:50.952148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.952075    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh" event={"ID":"b2f5d62b-8341-483f-8f30-1c96b01001c8","Type":"ContainerStarted","Data":"f37789667cd46340ec1842f3d4f121205477f1b9342551753f0f5c4af01f00f4"}
Nov 02 07:37:50.996224 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:50.996154    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj" podStartSLOduration=1.996135805 podStartE2EDuration="1.996135805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:50.992125172 +0000 UTC m=+821.682452137" watchObservedRunningTime="2026-11-02 07:37:50.996135805 +0000 UTC m=+821.686462771"
Nov 02 07:37:51.034455 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:51.034326    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh" podStartSLOduration=2.034311592 podStartE2EDuration="2.034311592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:37:51.032810216 +0000 UTC m=+821.723137208" watchObservedRunningTime="2026-11-02 07:37:51.034311592 +0000 UTC m=+821.724638556"
Nov 02 07:37:51.945387 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:51.945342    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:51.996911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:51.995512    4898 generic.go:356] "Generic (PLEG): container finished" podID="b2f5d62b-8341-483f-8f30-1c96b01001c8" containerID="745455174c9b1f34d264fbdc85ac3f9f33ff4f929f4905d745d9dc76bb57c1e0" exitCode=0
Nov 02 07:37:51.996911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:51.996006    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh" event={"ID":"b2f5d62b-8341-483f-8f30-1c96b01001c8","Type":"ContainerDied","Data":"745455174c9b1f34d264fbdc85ac3f9f33ff4f929f4905d745d9dc76bb57c1e0"}
Nov 02 07:37:53.443928 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:53.443898    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:53.575306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:53.575268    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-xks-ca\") pod \"b2f5d62b-8341-483f-8f30-1c96b01001c8\" (UID: \"b2f5d62b-8341-483f-8f30-1c96b01001c8\") "
Nov 02 07:37:53.585913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:53.575611    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f5d62b-8341-483f-8f30-1c96b01001c8-kube-api-access-msnt7\" (UniqueName: \"kubernetes.io/projected/b2f5d62b-8341-483f-8f30-1c96b01001c8-kube-api-access-msnt7\") pod \"b2f5d62b-8341-483f-8f30-1c96b01001c8\" (UID: \"b2f5d62b-8341-483f-8f30-1c96b01001c8\") "
Nov 02 07:37:53.586278 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:53.586253    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-openshift-ca\") pod \"b2f5d62b-8341-483f-8f30-1c96b01001c8\" (UID: \"b2f5d62b-8341-483f-8f30-1c96b01001c8\") "
Nov 02 07:37:53.609204 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:53.609113    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-openshift-ca" pod "b2f5d62b-8341-483f-8f30-1c96b01001c8" (UID: "b2f5d62b-8341-483f-8f30-1c96b01001c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:37:53.631911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:53.629135    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f5d62b-8341-483f-8f30-1c96b01001c8-kube-api-access-msnt7" pod "b2f5d62b-8341-483f-8f30-1c96b01001c8" (UID: "b2f5d62b-8341-483f-8f30-1c96b01001c8"). InnerVolumeSpecName "kube-api-access-msnt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:37:53.635618 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:53.635581    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-xks-ca" pod "b2f5d62b-8341-483f-8f30-1c96b01001c8" (UID: "b2f5d62b-8341-483f-8f30-1c96b01001c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:37:53.688868 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:53.688830    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:53.689044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:53.688873    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msnt7\" (UniqueName: \"kubernetes.io/projected/b2f5d62b-8341-483f-8f30-1c96b01001c8-kube-api-access-msnt7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:53.689044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:53.688909    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f5d62b-8341-483f-8f30-1c96b01001c8-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:37:54.025482 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:54.025444    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh" event={"ID":"b2f5d62b-8341-483f-8f30-1c96b01001c8","Type":"ContainerDied","Data":"f37789667cd46340ec1842f3d4f121205477f1b9342551753f0f5c4af01f00f4"}
Nov 02 07:37:54.025677 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:54.025492    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"
Nov 02 07:37:54.025677 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:54.025505    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f37789667cd46340ec1842f3d4f121205477f1b9342551753f0f5c4af01f00f4"
Nov 02 07:37:59.944044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:59.944008    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:37:59.957284 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:37:59.957259    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:38:00.083809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:00.083774    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:38:03.190305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.190263    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"]
Nov 02 07:38:03.191053 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.191034    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b2f5d62b-8341-483f-8f30-1c96b01001c8" containerName="revoke-keys"
Nov 02 07:38:03.191180 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.191054    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f5d62b-8341-483f-8f30-1c96b01001c8" containerName="revoke-keys"
Nov 02 07:38:03.191239 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.191179    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f5d62b-8341-483f-8f30-1c96b01001c8" containerName="revoke-keys"
Nov 02 07:38:03.195295 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.195273    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.207905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.207863    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"]
Nov 02 07:38:03.242600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.242563    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.242761 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.242606    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rdm\" (UniqueName: \"kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-kube-api-access-56rdm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.242761 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.242728    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.242875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.242851    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.243010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.242968    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.243010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.243012    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.243213 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.243131    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.243213 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.243164    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.243213 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.243203    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.344168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.344135    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.344342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.344177    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.344342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.344222    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.344342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.344304    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.344342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.344328    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.344503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.344362    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.344503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.344458    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.344503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.344483    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rdm\" (UniqueName: \"kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-kube-api-access-56rdm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.344646 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.344544    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.344646 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.344617    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.344750 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.344718    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.344803 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.344784    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.345185 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.345159    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.345185 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.345173    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.346529 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.346506    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.346908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.346876    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.368121 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.368096    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.368261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.368240    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rdm\" (UniqueName: \"kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-kube-api-access-56rdm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-68twk\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.513334 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.513245    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:03.829545 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:38:03.829516    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06eb45ea_2696_40be_9e1f_90bc0fe6fead.slice/crio-8660bf7001f1b416882e5be88b4c153a8ed75dc23f8d8aabe35bbc60ec4f659e WatchSource:0}: Error finding container 8660bf7001f1b416882e5be88b4c153a8ed75dc23f8d8aabe35bbc60ec4f659e: Status 404 returned error can't find the container with id 8660bf7001f1b416882e5be88b4c153a8ed75dc23f8d8aabe35bbc60ec4f659e
Nov 02 07:38:03.830703 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.830680    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"]
Nov 02 07:38:03.832268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.832234    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:38:03.832335 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.832298    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:38:03.832335 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:03.832327    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:38:04.106234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:04.106138    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" event={"ID":"06eb45ea-2696-40be-9e1f-90bc0fe6fead","Type":"ContainerStarted","Data":"f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879"}
Nov 02 07:38:04.106234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:04.106200    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" event={"ID":"06eb45ea-2696-40be-9e1f-90bc0fe6fead","Type":"ContainerStarted","Data":"8660bf7001f1b416882e5be88b4c153a8ed75dc23f8d8aabe35bbc60ec4f659e"}
Nov 02 07:38:04.151864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:04.151816    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" podStartSLOduration=1.151803082 podStartE2EDuration="1.151803082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:38:04.151567886 +0000 UTC m=+834.841894870" watchObservedRunningTime="2026-11-02 07:38:04.151803082 +0000 UTC m=+834.842130045"
Nov 02 07:38:04.513672 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:04.513640    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:04.515308 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:04.515282    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-68twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:38:04.515442 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:04.515338    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 07:38:05.517393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:05.513991    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-68twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:38:05.517393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:05.514062    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 07:38:06.514701 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:06.514235    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-68twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:38:06.514701 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:06.514294    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 07:38:07.518307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:07.518247    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-68twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:38:07.518911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:07.518855    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 07:38:07.788981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:07.788778    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded"
Nov 02 07:38:07.838914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:07.824029    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:38:07.900973 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:07.896003    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:38:08.518392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:08.518355    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-68twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:38:08.519293 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:08.518944    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 07:38:09.514209 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:09.514176    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-68twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:38:09.514375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:09.514228    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 07:38:10.513707 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:10.513664    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-68twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:38:10.514142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:10.513730    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 07:38:11.514143 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:11.514106    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-68twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 07:38:11.514586 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:11.514161    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 07:38:13.514242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:13.514195    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-68twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:38:13.514657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:13.514260    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:38:14.556080 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:14.556033    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-68twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:38:14.556577 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:14.556099    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:38:14.912950 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:14.912922    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:14.913669 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:14.913645    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:14.914334 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:14.914311    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:38:15.001561 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:38:15.001517    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:38:15.196310 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.196230    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"]
Nov 02 07:38:15.202701 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.202674    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.222317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.222285    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0fbc17\"/\"distinct-w0-0fbc17-kserve-self-signed-certs\""
Nov 02 07:38:15.228092 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.228062    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"]
Nov 02 07:38:15.300722 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.300692    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-home\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.300913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.300736    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698187ec-6d8a-4096-abef-c51a70654104-tls-certs\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.300913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.300759    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-tmp-dir\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.301056 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.300951    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c5nj\" (UniqueName: \"kubernetes.io/projected/698187ec-6d8a-4096-abef-c51a70654104-kube-api-access-5c5nj\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.301099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.301068    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-dshm\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.301218 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.301201    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-model-cache\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.409913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.408127    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-dshm\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.409913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.408294    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-model-cache\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.409913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.408348    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-home\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.409913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.408404    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698187ec-6d8a-4096-abef-c51a70654104-tls-certs\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.409913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.408465    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-tmp-dir\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.409913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.408533    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c5nj\" (UniqueName: \"kubernetes.io/projected/698187ec-6d8a-4096-abef-c51a70654104-kube-api-access-5c5nj\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.409913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.409073    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-home\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.409913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.409331    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-model-cache\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.410445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.410166    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-tmp-dir\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.413471 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.413442    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-dshm\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.423385 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.423343    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698187ec-6d8a-4096-abef-c51a70654104-tls-certs\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.432604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.432569    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c5nj\" (UniqueName: \"kubernetes.io/projected/698187ec-6d8a-4096-abef-c51a70654104-kube-api-access-5c5nj\") pod \"distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") " pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.526579 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.526502    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:15.830936 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:38:15.830903    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod698187ec_6d8a_4096_abef_c51a70654104.slice/crio-f968b29912bce37a9f9f24eab3817f3f2d573f6623b0b101c2bbcca5de7d8c62 WatchSource:0}: Error finding container f968b29912bce37a9f9f24eab3817f3f2d573f6623b0b101c2bbcca5de7d8c62: Status 404 returned error can't find the container with id f968b29912bce37a9f9f24eab3817f3f2d573f6623b0b101c2bbcca5de7d8c62
Nov 02 07:38:15.832313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:15.832290    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"]
Nov 02 07:38:16.309117 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:16.309078    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" event={"ID":"698187ec-6d8a-4096-abef-c51a70654104","Type":"ContainerStarted","Data":"6333e57018fb396231aa307a5f430c9f990f5a44109c5cc34c1eb74bf4509de8"}
Nov 02 07:38:16.309306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:16.309127    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" event={"ID":"698187ec-6d8a-4096-abef-c51a70654104","Type":"ContainerStarted","Data":"f968b29912bce37a9f9f24eab3817f3f2d573f6623b0b101c2bbcca5de7d8c62"}
Nov 02 07:38:16.351212 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:16.351151    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podStartSLOduration=1.35113689 podStartE2EDuration="1.35113689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:38:16.350077057 +0000 UTC m=+847.040404022" watchObservedRunningTime="2026-11-02 07:38:16.35113689 +0000 UTC m=+847.041463853"
Nov 02 07:38:16.527211 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:16.527173    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:25.527759 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:25.527676    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:25.540933 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:25.540908    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:26.399505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:26.399467    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:38:44.989376 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:38:44.989338    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:38:48.946961 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:48.946919    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"]
Nov 02 07:38:48.955973 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:48.955944    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:48.965190 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:48.965136    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"]
Nov 02 07:38:49.115824 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.115785    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.116038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.115855    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvf8b\" (UniqueName: \"kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-kube-api-access-fvf8b\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.116038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.115972    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.116038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.116002    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.116038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.116023    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.116232 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.116052    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.116232 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.116153    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.116232 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.116191    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.116232 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.116206    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.217424 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.217354    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvf8b\" (UniqueName: \"kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-kube-api-access-fvf8b\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.217424 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.217403    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.217424 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.217422    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.217696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.217446    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.217696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.217485    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.217696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.217521    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.217696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.217551    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.217696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.217575    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.217696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.217682    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.218084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.217931    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.218084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.218058    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.218084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.218077    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.218248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.218148    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.218306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.218267    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.220129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.220103    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.220339 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.220323    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.240585 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.240563    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.240724 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.240705    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvf8b\" (UniqueName: \"kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-kube-api-access-fvf8b\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-626zl\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.278177 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.278139    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:49.595393 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:38:49.595362    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a340f3b_b2c0_429d_b55a_4d0b5b6b4137.slice/crio-76293efae74c1e02fc240d86ebbc150a0e5f55a66b3eebee7adf078bb2514c7c WatchSource:0}: Error finding container 76293efae74c1e02fc240d86ebbc150a0e5f55a66b3eebee7adf078bb2514c7c: Status 404 returned error can't find the container with id 76293efae74c1e02fc240d86ebbc150a0e5f55a66b3eebee7adf078bb2514c7c
Nov 02 07:38:49.597658 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.597634    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"]
Nov 02 07:38:49.597785 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.597748    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:38:49.597848 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.597832    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:38:49.597913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:49.597874    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:38:50.604238 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:50.604197    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" event={"ID":"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137","Type":"ContainerStarted","Data":"6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965"}
Nov 02 07:38:50.604716 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:50.604294    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" event={"ID":"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137","Type":"ContainerStarted","Data":"76293efae74c1e02fc240d86ebbc150a0e5f55a66b3eebee7adf078bb2514c7c"}
Nov 02 07:38:50.686312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:50.686249    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" podStartSLOduration=2.68623315 podStartE2EDuration="2.68623315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:38:50.683300205 +0000 UTC m=+881.373627169" watchObservedRunningTime="2026-11-02 07:38:50.68623315 +0000 UTC m=+881.376560114"
Nov 02 07:38:51.280142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:51.280096    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:51.281796 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:51.281759    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-626zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 02 07:38:51.281952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:51.281824    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" podUID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 02 07:38:52.279312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:52.279272    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-626zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 02 07:38:52.279945 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:52.279911    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" podUID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 02 07:38:53.284389 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:53.284352    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-626zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 02 07:38:53.284974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:53.284419    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" podUID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 02 07:38:54.280909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:54.280285    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-626zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 02 07:38:54.280909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:54.280361    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" podUID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 02 07:38:55.286311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:55.286275    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-626zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 02 07:38:55.286745 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:55.286344    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" podUID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 02 07:38:56.279416 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:56.279374    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-626zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 02 07:38:56.279746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:56.279443    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" podUID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 02 07:38:57.283885 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:57.283848    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:57.284449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:57.284425    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:57.285206 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:57.285185    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:38:58.450451 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:58.450416    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"]
Nov 02 07:38:59.021982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.021941    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"]
Nov 02 07:38:59.026179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.026152    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:38:59.047029 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.046998    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"]
Nov 02 07:38:59.053742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.053717    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:38:59.054111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.054086    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:38:59.054235 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.054170    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:38:59.054313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.054298    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:38:59.157183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.157133    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9febe4d7-nns7k\" (UID: \"550bc00f-8763-442c-b9af-15abeba8f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:38:59.157380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.157312    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9febe4d7-nns7k\" (UID: \"550bc00f-8763-442c-b9af-15abeba8f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:38:59.157453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.157379    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplfp\" (UniqueName: \"kubernetes.io/projected/550bc00f-8763-442c-b9af-15abeba8f4c8-kube-api-access-dplfp\") pod \"maas-api-revoke-sub-9febe4d7-nns7k\" (UID: \"550bc00f-8763-442c-b9af-15abeba8f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:38:59.258958 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.258905    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9febe4d7-nns7k\" (UID: \"550bc00f-8763-442c-b9af-15abeba8f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:38:59.259139 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.258990    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplfp\" (UniqueName: \"kubernetes.io/projected/550bc00f-8763-442c-b9af-15abeba8f4c8-kube-api-access-dplfp\") pod \"maas-api-revoke-sub-9febe4d7-nns7k\" (UID: \"550bc00f-8763-442c-b9af-15abeba8f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:38:59.259139 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.259080    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9febe4d7-nns7k\" (UID: \"550bc00f-8763-442c-b9af-15abeba8f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:38:59.259937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.259871    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9febe4d7-nns7k\" (UID: \"550bc00f-8763-442c-b9af-15abeba8f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:38:59.261505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.261483    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9febe4d7-nns7k\" (UID: \"550bc00f-8763-442c-b9af-15abeba8f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:38:59.282629 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.282564    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplfp\" (UniqueName: \"kubernetes.io/projected/550bc00f-8763-442c-b9af-15abeba8f4c8-kube-api-access-dplfp\") pod \"maas-api-revoke-sub-9febe4d7-nns7k\" (UID: \"550bc00f-8763-442c-b9af-15abeba8f4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:38:59.344393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.344346    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:38:59.974426 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:38:59.974394    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"]
Nov 02 07:39:00.840931 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:00.839788    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k" event={"ID":"550bc00f-8763-442c-b9af-15abeba8f4c8","Type":"ContainerStarted","Data":"3fec609167b9e84faf32ee5d1d19981849ce5b25ef64d85c0665f5973f2f01af"}
Nov 02 07:39:00.840931 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:00.839863    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k" event={"ID":"550bc00f-8763-442c-b9af-15abeba8f4c8","Type":"ContainerStarted","Data":"914b74bb3a8f8aefa0f0e3d2b99546d50ece9b87c00571da1246bd580e852a50"}
Nov 02 07:39:01.827071 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:01.827022    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:01.904069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:01.904024    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:01.905251 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:01.905219    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:01.920044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:01.920011    4898 generic.go:356] "Generic (PLEG): container finished" podID="550bc00f-8763-442c-b9af-15abeba8f4c8" containerID="3fec609167b9e84faf32ee5d1d19981849ce5b25ef64d85c0665f5973f2f01af" exitCode=0
Nov 02 07:39:01.920342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:01.920316    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k" event={"ID":"550bc00f-8763-442c-b9af-15abeba8f4c8","Type":"ContainerDied","Data":"3fec609167b9e84faf32ee5d1d19981849ce5b25ef64d85c0665f5973f2f01af"}
Nov 02 07:39:02.088138 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:02.088033    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj" podUID="ae0eae37-8d3a-48da-ab8f-6b7bee43f176" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 02 07:39:02.322032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:02.321983    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:02.399117 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:02.398970    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:02.549511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:02.548972    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:02.832017 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:02.831968    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:02.833333 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:02.833309    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:39:02.903091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:02.903049    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:02.904445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:02.904409    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:02.904969 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:02.904949    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:39:03.157281 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.132531    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 07:39:03.278914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.213318    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj" podUID="ae0eae37-8d3a-48da-ab8f-6b7bee43f176" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:03.278914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.213943    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:39:03.466970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.429978    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:39:03.466970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.430651    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:39:03.655608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.632630    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:39:03.689064 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.689038    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:39:03.716911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.716716    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:39:03.739859 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.739605    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:39:03.913061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.913028    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:39:03.947572 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.947536    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-openshift-ca\") pod \"550bc00f-8763-442c-b9af-15abeba8f4c8\" (UID: \"550bc00f-8763-442c-b9af-15abeba8f4c8\") "
Nov 02 07:39:03.947760 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.947635    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-xks-ca\") pod \"550bc00f-8763-442c-b9af-15abeba8f4c8\" (UID: \"550bc00f-8763-442c-b9af-15abeba8f4c8\") "
Nov 02 07:39:03.947760 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.947720    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550bc00f-8763-442c-b9af-15abeba8f4c8-kube-api-access-dplfp\" (UniqueName: \"kubernetes.io/projected/550bc00f-8763-442c-b9af-15abeba8f4c8-kube-api-access-dplfp\") pod \"550bc00f-8763-442c-b9af-15abeba8f4c8\" (UID: \"550bc00f-8763-442c-b9af-15abeba8f4c8\") "
Nov 02 07:39:03.947920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.947871    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-openshift-ca" pod "550bc00f-8763-442c-b9af-15abeba8f4c8" (UID: "550bc00f-8763-442c-b9af-15abeba8f4c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:03.948182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.948155    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:03.949721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.949689    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-xks-ca" pod "550bc00f-8763-442c-b9af-15abeba8f4c8" (UID: "550bc00f-8763-442c-b9af-15abeba8f4c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:39:03.949826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:03.949720    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550bc00f-8763-442c-b9af-15abeba8f4c8-kube-api-access-dplfp" pod "550bc00f-8763-442c-b9af-15abeba8f4c8" (UID: "550bc00f-8763-442c-b9af-15abeba8f4c8"). InnerVolumeSpecName "kube-api-access-dplfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:04.049602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:04.049527    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550bc00f-8763-442c-b9af-15abeba8f4c8-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:04.049602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:04.049555    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dplfp\" (UniqueName: \"kubernetes.io/projected/550bc00f-8763-442c-b9af-15abeba8f4c8-kube-api-access-dplfp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:04.255711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:04.255677    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"
Nov 02 07:39:04.255711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:04.255691    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k" event={"ID":"550bc00f-8763-442c-b9af-15abeba8f4c8","Type":"ContainerDied","Data":"914b74bb3a8f8aefa0f0e3d2b99546d50ece9b87c00571da1246bd580e852a50"}
Nov 02 07:39:04.255998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:04.255752    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="914b74bb3a8f8aefa0f0e3d2b99546d50ece9b87c00571da1246bd580e852a50"
Nov 02 07:39:09.692103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:09.692072    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:39:09.702492 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:09.702466    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:39:13.472278 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.472203    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj" podUID="ae0eae37-8d3a-48da-ab8f-6b7bee43f176" containerName="main" containerID="cri-o://931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a" gracePeriod=45
Nov 02 07:39:13.759315 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.759284    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:39:13.891076 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.891037    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-model-cache\") pod \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") "
Nov 02 07:39:13.891269 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.891098    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-kube-api-access-tgdcb\" (UniqueName: \"kubernetes.io/projected/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-kube-api-access-tgdcb\") pod \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") "
Nov 02 07:39:13.891269 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.891204    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-home\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-home\") pod \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") "
Nov 02 07:39:13.891393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.891291    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tmp-dir\") pod \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") "
Nov 02 07:39:13.891393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.891329    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-model-cache" pod "ae0eae37-8d3a-48da-ab8f-6b7bee43f176" (UID: "ae0eae37-8d3a-48da-ab8f-6b7bee43f176"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:39:13.891393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.891371    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-dshm\") pod \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") "
Nov 02 07:39:13.891564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.891412    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tls-certs\") pod \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\" (UID: \"ae0eae37-8d3a-48da-ab8f-6b7bee43f176\") "
Nov 02 07:39:13.891564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.891552    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-home" pod "ae0eae37-8d3a-48da-ab8f-6b7bee43f176" (UID: "ae0eae37-8d3a-48da-ab8f-6b7bee43f176"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:39:13.891938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.891916    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:13.892047 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.891940    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:13.892104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.892083    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tmp-dir" pod "ae0eae37-8d3a-48da-ab8f-6b7bee43f176" (UID: "ae0eae37-8d3a-48da-ab8f-6b7bee43f176"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:39:13.893989 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.893961    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tls-certs" pod "ae0eae37-8d3a-48da-ab8f-6b7bee43f176" (UID: "ae0eae37-8d3a-48da-ab8f-6b7bee43f176"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:39:13.894221 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.894195    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-kube-api-access-tgdcb" pod "ae0eae37-8d3a-48da-ab8f-6b7bee43f176" (UID: "ae0eae37-8d3a-48da-ab8f-6b7bee43f176"). InnerVolumeSpecName "kube-api-access-tgdcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:13.894317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.894299    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-dshm" pod "ae0eae37-8d3a-48da-ab8f-6b7bee43f176" (UID: "ae0eae37-8d3a-48da-ab8f-6b7bee43f176"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:39:13.930585 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.930561    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"]
Nov 02 07:39:13.931165 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.931122    4898 cpu_manager.go:414] "containerMap: removing container" podUID="550bc00f-8763-442c-b9af-15abeba8f4c8" containerName="revoke-keys"
Nov 02 07:39:13.931165 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.931145    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550bc00f-8763-442c-b9af-15abeba8f4c8" containerName="revoke-keys"
Nov 02 07:39:13.931350 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.931191    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ae0eae37-8d3a-48da-ab8f-6b7bee43f176" containerName="main"
Nov 02 07:39:13.931350 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.931199    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae0eae37-8d3a-48da-ab8f-6b7bee43f176" containerName="main"
Nov 02 07:39:13.931350 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.931309    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="550bc00f-8763-442c-b9af-15abeba8f4c8" containerName="revoke-keys"
Nov 02 07:39:13.931350 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.931326    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae0eae37-8d3a-48da-ab8f-6b7bee43f176" containerName="main"
Nov 02 07:39:13.935332 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.935316    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:13.952017 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.951640    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-337764\"/\"distinct-w2-337764-kserve-self-signed-certs\""
Nov 02 07:39:13.956926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.956197    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"]
Nov 02 07:39:13.993614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.993533    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgdcb\" (UniqueName: \"kubernetes.io/projected/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-kube-api-access-tgdcb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:13.993614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.993565    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:13.993614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.993580    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:13.993614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.993591    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0eae37-8d3a-48da-ab8f-6b7bee43f176-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:13.996171 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:13.996112    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"]
Nov 02 07:39:14.010092 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.010060    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:14.017660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.017629    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"]
Nov 02 07:39:14.026903 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.026864    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:39:14.027015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.026877    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:39:14.051804 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.051777    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:39:14.052093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.052072    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:39:14.094696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.094664    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fabe65f5-gv6mb\" (UID: \"328ac24d-ccf6-4faf-93d7-42aa4238923e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:14.094857 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.094709    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8jzv\" (UniqueName: \"kubernetes.io/projected/328ac24d-ccf6-4faf-93d7-42aa4238923e-kube-api-access-k8jzv\") pod \"maas-api-revoke-sub-fabe65f5-gv6mb\" (UID: \"328ac24d-ccf6-4faf-93d7-42aa4238923e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:14.094857 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.094806    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tls-certs\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.095034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.094915    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-home\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.095099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.095054    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fabe65f5-gv6mb\" (UID: \"328ac24d-ccf6-4faf-93d7-42aa4238923e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:14.095099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.095087    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tmp-dir\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.095208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.095133    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-model-cache\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.095208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.095183    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts6jr\" (UniqueName: \"kubernetes.io/projected/12c3003c-49e7-44cb-abb4-06ddddcbfafe-kube-api-access-ts6jr\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.095311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.095282    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-dshm\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.196711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.196676    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fabe65f5-gv6mb\" (UID: \"328ac24d-ccf6-4faf-93d7-42aa4238923e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:14.196711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.196719    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tmp-dir\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.196998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.196758    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-model-cache\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.196998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.196782    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts6jr\" (UniqueName: \"kubernetes.io/projected/12c3003c-49e7-44cb-abb4-06ddddcbfafe-kube-api-access-ts6jr\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.196998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.196854    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-dshm\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.197239 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.197208    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tmp-dir\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.197302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.197243    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-model-cache\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.197354 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.197310    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fabe65f5-gv6mb\" (UID: \"328ac24d-ccf6-4faf-93d7-42aa4238923e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:14.197414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.197367    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8jzv\" (UniqueName: \"kubernetes.io/projected/328ac24d-ccf6-4faf-93d7-42aa4238923e-kube-api-access-k8jzv\") pod \"maas-api-revoke-sub-fabe65f5-gv6mb\" (UID: \"328ac24d-ccf6-4faf-93d7-42aa4238923e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:14.197465 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.197442    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tls-certs\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.197551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.197534    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-home\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.197745 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.197722    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fabe65f5-gv6mb\" (UID: \"328ac24d-ccf6-4faf-93d7-42aa4238923e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:14.197946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.197884    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-home\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.200192 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.200161    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tls-certs\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.200192 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.200185    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-dshm\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.200386 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.200284    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fabe65f5-gv6mb\" (UID: \"328ac24d-ccf6-4faf-93d7-42aa4238923e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:14.220823 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.220793    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts6jr\" (UniqueName: \"kubernetes.io/projected/12c3003c-49e7-44cb-abb4-06ddddcbfafe-kube-api-access-ts6jr\") pod \"distinct-w2-337764-kserve-67cf9d8469-xhbsq\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") " pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.220976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.220855    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8jzv\" (UniqueName: \"kubernetes.io/projected/328ac24d-ccf6-4faf-93d7-42aa4238923e-kube-api-access-k8jzv\") pod \"maas-api-revoke-sub-fabe65f5-gv6mb\" (UID: \"328ac24d-ccf6-4faf-93d7-42aa4238923e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:14.252402 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.252318    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:14.325398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.325054    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:14.333111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.333083    4898 generic.go:356] "Generic (PLEG): container finished" podID="ae0eae37-8d3a-48da-ab8f-6b7bee43f176" containerID="931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a" exitCode=0
Nov 02 07:39:14.333218 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.333154    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"
Nov 02 07:39:14.333300 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.333152    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj" event={"ID":"ae0eae37-8d3a-48da-ab8f-6b7bee43f176","Type":"ContainerDied","Data":"931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a"}
Nov 02 07:39:14.333355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.333320    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj" event={"ID":"ae0eae37-8d3a-48da-ab8f-6b7bee43f176","Type":"ContainerDied","Data":"14a00f25407c70871e324ba83f965aece8c9ba8d15257167dc684e9f39f50445"}
Nov 02 07:39:14.333467 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.333448    4898 scope.go:122] "RemoveContainer" containerID="931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a"
Nov 02 07:39:14.349807 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.349565    4898 scope.go:122] "RemoveContainer" containerID="931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a"
Nov 02 07:39:14.350118 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:39:14.350053    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a\": container with ID starting with 931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a not found: ID does not exist" containerID="931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a"
Nov 02 07:39:14.350118 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.350092    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a"} err="failed to get container status \"931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a\": rpc error: code = NotFound desc = could not find container \"931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a\": container with ID starting with 931017ae9817c10291b400992859214432c1767691f921e884601d2a9fa7f38a not found: ID does not exist"
Nov 02 07:39:14.390946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.390726    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"]
Nov 02 07:39:14.398927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.398024    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-337764/e2e-conflict-a-d0b940c5-kserve-5b869b4bfb-6gswj"]
Nov 02 07:39:14.553585 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:39:14.553550    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12c3003c_49e7_44cb_abb4_06ddddcbfafe.slice/crio-81598b8dcb6ddc54d36e9dd1a47f5f285449a9996bc2538390cfe472fc82234b WatchSource:0}: Error finding container 81598b8dcb6ddc54d36e9dd1a47f5f285449a9996bc2538390cfe472fc82234b: Status 404 returned error can't find the container with id 81598b8dcb6ddc54d36e9dd1a47f5f285449a9996bc2538390cfe472fc82234b
Nov 02 07:39:14.555517 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.555481    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"]
Nov 02 07:39:14.646442 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:39:14.646418    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328ac24d_ccf6_4faf_93d7_42aa4238923e.slice/crio-1c2fbbb47f80d5ee26f58f1e8ce29bc09428b90eac19c97903b8892f7af49e9c WatchSource:0}: Error finding container 1c2fbbb47f80d5ee26f58f1e8ce29bc09428b90eac19c97903b8892f7af49e9c: Status 404 returned error can't find the container with id 1c2fbbb47f80d5ee26f58f1e8ce29bc09428b90eac19c97903b8892f7af49e9c
Nov 02 07:39:14.647982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:14.647959    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"]
Nov 02 07:39:14.993276 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:39:14.993241    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:39:15.345748 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.345707    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb" event={"ID":"328ac24d-ccf6-4faf-93d7-42aa4238923e","Type":"ContainerStarted","Data":"b3e7cd04f47d478cd6a4e4c335025e8e50e7c8603d0bebf14bf27860e7fa22d0"}
Nov 02 07:39:15.345993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.345767    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb" event={"ID":"328ac24d-ccf6-4faf-93d7-42aa4238923e","Type":"ContainerStarted","Data":"1c2fbbb47f80d5ee26f58f1e8ce29bc09428b90eac19c97903b8892f7af49e9c"}
Nov 02 07:39:15.347640 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.347581    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" event={"ID":"12c3003c-49e7-44cb-abb4-06ddddcbfafe","Type":"ContainerStarted","Data":"447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490"}
Nov 02 07:39:15.347640 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.347619    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" event={"ID":"12c3003c-49e7-44cb-abb4-06ddddcbfafe","Type":"ContainerStarted","Data":"81598b8dcb6ddc54d36e9dd1a47f5f285449a9996bc2538390cfe472fc82234b"}
Nov 02 07:39:15.348520 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.348494    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"]
Nov 02 07:39:15.352541 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.352520    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.379458 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.379421    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"]
Nov 02 07:39:15.436353 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.436292    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podStartSLOduration=2.436272304 podStartE2EDuration="2.436272304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:15.434639328 +0000 UTC m=+906.124966292" watchObservedRunningTime="2026-11-02 07:39:15.436272304 +0000 UTC m=+906.126599269"
Nov 02 07:39:15.439330 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.439246    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb" podStartSLOduration=2.439228884 podStartE2EDuration="2.439228884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:15.393356044 +0000 UTC m=+906.083683011" watchObservedRunningTime="2026-11-02 07:39:15.439228884 +0000 UTC m=+906.129555846"
Nov 02 07:39:15.468347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.468311    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae0eae37-8d3a-48da-ab8f-6b7bee43f176" path="/var/lib/kubelet/pods/ae0eae37-8d3a-48da-ab8f-6b7bee43f176/volumes"
Nov 02 07:39:15.516739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.516699    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.516953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.516749    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.516953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.516778    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.516953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.516922    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spqft\" (UniqueName: \"kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-kube-api-access-spqft\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.517125 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.516978    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.517179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.517134    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.517432 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.517231    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.517432 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.517349    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.517561 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.517448    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8912595-8874-4a75-b536-eeecdbfcc40a-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.620530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.618692    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.620530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.618812    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.620530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.618927    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.620530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.619034    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.620530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.619095    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8912595-8874-4a75-b536-eeecdbfcc40a-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.620530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.619144    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.620530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.619184    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.620530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.619229    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.620530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.619266    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spqft\" (UniqueName: \"kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-kube-api-access-spqft\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.620530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.620036    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.621519 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.620651    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8912595-8874-4a75-b536-eeecdbfcc40a-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.621519 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.620736    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.621519 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.621269    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.621519 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.621493    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.628055 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.628005    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.629060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.629032    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.642439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.642398    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spqft\" (UniqueName: \"kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-kube-api-access-spqft\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.643303 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.643281    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:15.674395 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:15.674361    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:16.007727 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:39:16.007693    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8912595_8874_4a75_b536_eeecdbfcc40a.slice/crio-cb46723595005ad1f2d00d0f41f0686fff69f361ba929c4dc2507f7345cca9ac WatchSource:0}: Error finding container cb46723595005ad1f2d00d0f41f0686fff69f361ba929c4dc2507f7345cca9ac: Status 404 returned error can't find the container with id cb46723595005ad1f2d00d0f41f0686fff69f361ba929c4dc2507f7345cca9ac
Nov 02 07:39:16.007877 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:16.007769    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"]
Nov 02 07:39:16.010781 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:16.010737    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:16.010876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:16.010838    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:16.010971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:16.010918    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:16.254081 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:16.253977    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:16.401135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:16.401087    4898 generic.go:356] "Generic (PLEG): container finished" podID="328ac24d-ccf6-4faf-93d7-42aa4238923e" containerID="b3e7cd04f47d478cd6a4e4c335025e8e50e7c8603d0bebf14bf27860e7fa22d0" exitCode=0
Nov 02 07:39:16.401480 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:16.401454    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb" event={"ID":"328ac24d-ccf6-4faf-93d7-42aa4238923e","Type":"ContainerDied","Data":"b3e7cd04f47d478cd6a4e4c335025e8e50e7c8603d0bebf14bf27860e7fa22d0"}
Nov 02 07:39:16.427926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:16.427121    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" event={"ID":"b8912595-8874-4a75-b536-eeecdbfcc40a","Type":"ContainerStarted","Data":"6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e"}
Nov 02 07:39:16.427926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:16.427183    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" event={"ID":"b8912595-8874-4a75-b536-eeecdbfcc40a","Type":"ContainerStarted","Data":"cb46723595005ad1f2d00d0f41f0686fff69f361ba929c4dc2507f7345cca9ac"}
Nov 02 07:39:16.534096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:16.533977    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" podStartSLOduration=1.5339576639999999 podStartE2EDuration="1.533957664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:16.532095876 +0000 UTC m=+907.222422870" watchObservedRunningTime="2026-11-02 07:39:16.533957664 +0000 UTC m=+907.224284629"
Nov 02 07:39:16.685775 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:16.685734    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:17.743436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:17.741396    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 02 07:39:17.743436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:17.741462    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 02 07:39:18.173600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.173552    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:18.251913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.238566    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-xks-ca\") pod \"328ac24d-ccf6-4faf-93d7-42aa4238923e\" (UID: \"328ac24d-ccf6-4faf-93d7-42aa4238923e\") "
Nov 02 07:39:18.251913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.238655    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/328ac24d-ccf6-4faf-93d7-42aa4238923e-kube-api-access-k8jzv\" (UniqueName: \"kubernetes.io/projected/328ac24d-ccf6-4faf-93d7-42aa4238923e-kube-api-access-k8jzv\") pod \"328ac24d-ccf6-4faf-93d7-42aa4238923e\" (UID: \"328ac24d-ccf6-4faf-93d7-42aa4238923e\") "
Nov 02 07:39:18.251913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.238726    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-openshift-ca\") pod \"328ac24d-ccf6-4faf-93d7-42aa4238923e\" (UID: \"328ac24d-ccf6-4faf-93d7-42aa4238923e\") "
Nov 02 07:39:18.273159 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.273113    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-openshift-ca" pod "328ac24d-ccf6-4faf-93d7-42aa4238923e" (UID: "328ac24d-ccf6-4faf-93d7-42aa4238923e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:18.340968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.340356    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:18.598668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.598588    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-xks-ca" pod "328ac24d-ccf6-4faf-93d7-42aa4238923e" (UID: "328ac24d-ccf6-4faf-93d7-42aa4238923e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:39:18.607980 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.599231    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328ac24d-ccf6-4faf-93d7-42aa4238923e-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:18.645386 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.645052    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328ac24d-ccf6-4faf-93d7-42aa4238923e-kube-api-access-k8jzv" pod "328ac24d-ccf6-4faf-93d7-42aa4238923e" (UID: "328ac24d-ccf6-4faf-93d7-42aa4238923e"). InnerVolumeSpecName "kube-api-access-k8jzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:18.701205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.701165    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8jzv\" (UniqueName: \"kubernetes.io/projected/328ac24d-ccf6-4faf-93d7-42aa4238923e-kube-api-access-k8jzv\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:18.702202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.701760    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 02 07:39:18.702457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.702426    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 02 07:39:18.765389 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.763748    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"
Nov 02 07:39:18.767962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.763769    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb" event={"ID":"328ac24d-ccf6-4faf-93d7-42aa4238923e","Type":"ContainerDied","Data":"1c2fbbb47f80d5ee26f58f1e8ce29bc09428b90eac19c97903b8892f7af49e9c"}
Nov 02 07:39:18.768146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.768125    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c2fbbb47f80d5ee26f58f1e8ce29bc09428b90eac19c97903b8892f7af49e9c"
Nov 02 07:39:18.785584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.785298    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:18.900380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:18.899163    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:19.106007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:19.105962    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 07:39:19.503808 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:19.501908    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:19.638378 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:19.638332    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:19.679604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:19.679566    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 02 07:39:19.690007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:19.689964    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 02 07:39:19.900178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:19.900042    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 07:39:19.901911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:19.901843    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:39:19.931927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:19.927975    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:19.931927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:19.928351    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:39:20.106662 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:20.106607    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:20.107639 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:20.107615    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:39:20.227999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:20.191000    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:20.517977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:20.505157    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:20.517977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:20.505800    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:39:20.803552 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:20.803449    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:20.804643 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:20.804614    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:39:21.251923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:20.807541    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 02 07:39:21.251923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:21.248876    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 02 07:39:21.251923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:20.928800    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:39:21.251923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:20.807598    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:39:21.251923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:20.945212    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:21.251923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:21.116710    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:21.380750 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:21.380573    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:39:21.407766 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:21.407729    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:39:21.416926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:21.416902    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:39:21.428335 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:21.426171    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:39:21.436375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:21.436345    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:39:21.686934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:21.677720    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 02 07:39:21.686934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:21.677784    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 02 07:39:22.690322 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:22.686315    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 02 07:39:22.690322 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:22.686380    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 02 07:39:23.678354 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:23.678310    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 02 07:39:23.679533 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:23.679495    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 02 07:39:24.253478 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:24.253444    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:24.281091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:24.281056    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:24.330964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:24.330935    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:39:24.674965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:24.674932    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 02 07:39:24.675134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:24.675018    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 02 07:39:25.679706 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.679671    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:25.680141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.680095    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:25.680932 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.680884    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 07:39:25.868050 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.868017    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"]
Nov 02 07:39:25.868844 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.868825    4898 cpu_manager.go:414] "containerMap: removing container" podUID="328ac24d-ccf6-4faf-93d7-42aa4238923e" containerName="revoke-keys"
Nov 02 07:39:25.868942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.868850    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="328ac24d-ccf6-4faf-93d7-42aa4238923e" containerName="revoke-keys"
Nov 02 07:39:25.869045 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.869033    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="328ac24d-ccf6-4faf-93d7-42aa4238923e" containerName="revoke-keys"
Nov 02 07:39:25.874328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.874307    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:25.889205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.889179    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:39:25.889476 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.889452    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:39:25.889575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.889460    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:39:25.889575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.889487    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:39:25.889957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.889934    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"]
Nov 02 07:39:25.917441 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.917404    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b42fa15d-7nljl\" (UID: \"c9877ef1-2f01-46bb-af62-6753549ce0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:25.917764 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.917738    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b42fa15d-7nljl\" (UID: \"c9877ef1-2f01-46bb-af62-6753549ce0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:25.918054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:25.918029    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdrrb\" (UniqueName: \"kubernetes.io/projected/c9877ef1-2f01-46bb-af62-6753549ce0ea-kube-api-access-cdrrb\") pod \"maas-api-revoke-sub-b42fa15d-7nljl\" (UID: \"c9877ef1-2f01-46bb-af62-6753549ce0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:26.019842 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:26.019748    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b42fa15d-7nljl\" (UID: \"c9877ef1-2f01-46bb-af62-6753549ce0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:26.019842 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:26.019798    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdrrb\" (UniqueName: \"kubernetes.io/projected/c9877ef1-2f01-46bb-af62-6753549ce0ea-kube-api-access-cdrrb\") pod \"maas-api-revoke-sub-b42fa15d-7nljl\" (UID: \"c9877ef1-2f01-46bb-af62-6753549ce0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:26.020054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:26.019866    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b42fa15d-7nljl\" (UID: \"c9877ef1-2f01-46bb-af62-6753549ce0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:26.020668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:26.020638    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b42fa15d-7nljl\" (UID: \"c9877ef1-2f01-46bb-af62-6753549ce0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:26.022290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:26.022271    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b42fa15d-7nljl\" (UID: \"c9877ef1-2f01-46bb-af62-6753549ce0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:26.043081 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:26.043060    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdrrb\" (UniqueName: \"kubernetes.io/projected/c9877ef1-2f01-46bb-af62-6753549ce0ea-kube-api-access-cdrrb\") pod \"maas-api-revoke-sub-b42fa15d-7nljl\" (UID: \"c9877ef1-2f01-46bb-af62-6753549ce0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:26.189748 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:26.189716    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:26.506021 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:39:26.505984    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9877ef1_2f01_46bb_af62_6753549ce0ea.slice/crio-ec1ba3257e5d0c4cc7a34137a06faf735e52a953a7ad7bce00e409f1b40f4686 WatchSource:0}: Error finding container ec1ba3257e5d0c4cc7a34137a06faf735e52a953a7ad7bce00e409f1b40f4686: Status 404 returned error can't find the container with id ec1ba3257e5d0c4cc7a34137a06faf735e52a953a7ad7bce00e409f1b40f4686
Nov 02 07:39:26.507045 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:26.507010    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"]
Nov 02 07:39:27.343405 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:27.343372    4898 generic.go:356] "Generic (PLEG): container finished" podID="c9877ef1-2f01-46bb-af62-6753549ce0ea" containerID="26d20e5cda9383ac33df5f57328ee0e20af61db0b7805833776f16aca92aa3d8" exitCode=0
Nov 02 07:39:27.343762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:27.343504    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl" event={"ID":"c9877ef1-2f01-46bb-af62-6753549ce0ea","Type":"ContainerDied","Data":"26d20e5cda9383ac33df5f57328ee0e20af61db0b7805833776f16aca92aa3d8"}
Nov 02 07:39:27.343762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:27.343565    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl" event={"ID":"c9877ef1-2f01-46bb-af62-6753549ce0ea","Type":"ContainerStarted","Data":"ec1ba3257e5d0c4cc7a34137a06faf735e52a953a7ad7bce00e409f1b40f4686"}
Nov 02 07:39:28.529249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.529221    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:28.652459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.652435    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9877ef1-2f01-46bb-af62-6753549ce0ea-kube-api-access-cdrrb\" (UniqueName: \"kubernetes.io/projected/c9877ef1-2f01-46bb-af62-6753549ce0ea-kube-api-access-cdrrb\") pod \"c9877ef1-2f01-46bb-af62-6753549ce0ea\" (UID: \"c9877ef1-2f01-46bb-af62-6753549ce0ea\") "
Nov 02 07:39:28.652618 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.652576    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-xks-ca\") pod \"c9877ef1-2f01-46bb-af62-6753549ce0ea\" (UID: \"c9877ef1-2f01-46bb-af62-6753549ce0ea\") "
Nov 02 07:39:28.652618 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.652610    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-openshift-ca\") pod \"c9877ef1-2f01-46bb-af62-6753549ce0ea\" (UID: \"c9877ef1-2f01-46bb-af62-6753549ce0ea\") "
Nov 02 07:39:28.653060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.653019    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-openshift-ca" pod "c9877ef1-2f01-46bb-af62-6753549ce0ea" (UID: "c9877ef1-2f01-46bb-af62-6753549ce0ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:28.654647 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.654623    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9877ef1-2f01-46bb-af62-6753549ce0ea-kube-api-access-cdrrb" pod "c9877ef1-2f01-46bb-af62-6753549ce0ea" (UID: "c9877ef1-2f01-46bb-af62-6753549ce0ea"). InnerVolumeSpecName "kube-api-access-cdrrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:28.655609 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.655585    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-xks-ca" pod "c9877ef1-2f01-46bb-af62-6753549ce0ea" (UID: "c9877ef1-2f01-46bb-af62-6753549ce0ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:39:28.754046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.754012    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:28.754046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.754041    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9877ef1-2f01-46bb-af62-6753549ce0ea-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:28.754046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.754053    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdrrb\" (UniqueName: \"kubernetes.io/projected/c9877ef1-2f01-46bb-af62-6753549ce0ea-kube-api-access-cdrrb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:28.931124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.931042    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"]
Nov 02 07:39:28.931947 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.931921    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c9877ef1-2f01-46bb-af62-6753549ce0ea" containerName="revoke-keys"
Nov 02 07:39:28.931947 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.931946    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9877ef1-2f01-46bb-af62-6753549ce0ea" containerName="revoke-keys"
Nov 02 07:39:28.932137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.932125    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9877ef1-2f01-46bb-af62-6753549ce0ea" containerName="revoke-keys"
Nov 02 07:39:28.936284 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.936264    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:28.960784 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:28.960751    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"]
Nov 02 07:39:29.058060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.058019    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc472\" (UniqueName: \"kubernetes.io/projected/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-kube-api-access-rc472\") pod \"maas-api-revoke-sub-1b6e2994-wpdch\" (UID: \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:29.058283 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.058156    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6e2994-wpdch\" (UID: \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:29.058283 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.058199    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6e2994-wpdch\" (UID: \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:29.159425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.159379    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc472\" (UniqueName: \"kubernetes.io/projected/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-kube-api-access-rc472\") pod \"maas-api-revoke-sub-1b6e2994-wpdch\" (UID: \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:29.159619 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.159484    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6e2994-wpdch\" (UID: \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:29.159619 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.159508    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6e2994-wpdch\" (UID: \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:29.160358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.160328    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6e2994-wpdch\" (UID: \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:29.162310 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.162281    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6e2994-wpdch\" (UID: \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:29.184746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.184660    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc472\" (UniqueName: \"kubernetes.io/projected/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-kube-api-access-rc472\") pod \"maas-api-revoke-sub-1b6e2994-wpdch\" (UID: \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:29.255328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.255079    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:29.372420 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.371918    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"
Nov 02 07:39:29.372420 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.371910    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl" event={"ID":"c9877ef1-2f01-46bb-af62-6753549ce0ea","Type":"ContainerDied","Data":"ec1ba3257e5d0c4cc7a34137a06faf735e52a953a7ad7bce00e409f1b40f4686"}
Nov 02 07:39:29.372420 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.372047    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec1ba3257e5d0c4cc7a34137a06faf735e52a953a7ad7bce00e409f1b40f4686"
Nov 02 07:39:29.597139 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:39:29.597095    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9f27f0_ab89_41d1_bb4a_0f620923b4d5.slice/crio-20604d18b509a45034516013d34332333fddb7464c5f9700ca35fd14c73b93d3 WatchSource:0}: Error finding container 20604d18b509a45034516013d34332333fddb7464c5f9700ca35fd14c73b93d3: Status 404 returned error can't find the container with id 20604d18b509a45034516013d34332333fddb7464c5f9700ca35fd14c73b93d3
Nov 02 07:39:29.598839 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:29.598815    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"]
Nov 02 07:39:30.414921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.414867    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"]
Nov 02 07:39:30.532913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.529229    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"]
Nov 02 07:39:30.532913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.529278    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch" event={"ID":"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5","Type":"ContainerStarted","Data":"5ae622b894ded9b6055fb218c77d21a863e35825ead4b1bab08dc99ccac773c4"}
Nov 02 07:39:30.532913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.529317    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch" event={"ID":"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5","Type":"ContainerStarted","Data":"20604d18b509a45034516013d34332333fddb7464c5f9700ca35fd14c73b93d3"}
Nov 02 07:39:30.532913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.529415    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.632963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.632374    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.632963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.632431    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.632963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.632460    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0738e017-3fb4-4004-aa44-671d2a0a4400-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.632963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.632558    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.632963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.632602    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zw5f\" (UniqueName: \"kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-kube-api-access-2zw5f\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.632963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.632674    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.632963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.632718    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.632963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.632762    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.632963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.632814    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.734646 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.734611    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.734942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.734918    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zw5f\" (UniqueName: \"kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-kube-api-access-2zw5f\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.741054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.741027    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.741362 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.741248    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.746179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.746060    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.746364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.746345    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.760439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.748292    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.760622 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.753717    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.760770 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.760749    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.763920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.760938    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.764110 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.764091    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0738e017-3fb4-4004-aa44-671d2a0a4400-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.798908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.798851    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0738e017-3fb4-4004-aa44-671d2a0a4400-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.803817 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.803761    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch" podStartSLOduration=2.803742899 podStartE2EDuration="2.803742899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:30.591220338 +0000 UTC m=+921.281547305" watchObservedRunningTime="2026-11-02 07:39:30.803742899 +0000 UTC m=+921.494069864"
Nov 02 07:39:30.829553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.804334    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.829763 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.804581    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.829916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.826995    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.892006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.882469    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.893728 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.893667    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:30.894584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:30.894560    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zw5f\" (UniqueName: \"kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-kube-api-access-2zw5f\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:31.035171 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:31.035131    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:31.266110 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:31.265605    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:31.303582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:31.298826    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:31.433059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:31.428049    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:31.526981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:31.523787    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"]
Nov 02 07:39:31.526981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:31.524658    4898 generic.go:356] "Generic (PLEG): container finished" podID="cc9f27f0-ab89-41d1-bb4a-0f620923b4d5" containerID="5ae622b894ded9b6055fb218c77d21a863e35825ead4b1bab08dc99ccac773c4" exitCode=0
Nov 02 07:39:31.526981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:31.524789    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch" event={"ID":"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5","Type":"ContainerDied","Data":"5ae622b894ded9b6055fb218c77d21a863e35825ead4b1bab08dc99ccac773c4"}
Nov 02 07:39:31.526981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:31.525520    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:31.526981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:31.525593    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:31.526981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:31.525631    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:31.710935 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:31.709381    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:39:32.302878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:32.301203    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 07:39:32.302878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:32.301967    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:39:32.780047 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:32.779106    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:32.780582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:32.780422    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:39:32.857581 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:32.857540    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" event={"ID":"0738e017-3fb4-4004-aa44-671d2a0a4400","Type":"ContainerStarted","Data":"492134a347ade079655780fde0d851f7ce36f495a050eb533c2ec6930e3f6f6d"}
Nov 02 07:39:32.857866 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:32.857840    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" event={"ID":"0738e017-3fb4-4004-aa44-671d2a0a4400","Type":"ContainerStarted","Data":"584011f90dfedca24c5d8982b7a6d877308c5a662fbd3520ee4c4e00823b8007"}
Nov 02 07:39:32.975794 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:32.975729    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" podStartSLOduration=2.975705432 podStartE2EDuration="2.975705432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:32.94649586 +0000 UTC m=+923.636822828" watchObservedRunningTime="2026-11-02 07:39:32.975705432 +0000 UTC m=+923.666032397"
Nov 02 07:39:33.055974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:33.042371    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:33.055974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:33.042513    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 07:39:33.055974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:33.042563    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 07:39:33.302020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:33.301969    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:33.369027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:33.367899    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:39:33.868915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:33.853123    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:39:34.153107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.152134    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 07:39:34.153107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.152203    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 07:39:34.237147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.237117    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:34.277051 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.277005    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:39:34.363142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.363099    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-openshift-ca\") pod \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\" (UID: \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\") "
Nov 02 07:39:34.363996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.363791    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-openshift-ca" pod "cc9f27f0-ab89-41d1-bb4a-0f620923b4d5" (UID: "cc9f27f0-ab89-41d1-bb4a-0f620923b4d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:34.364525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.364493    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-xks-ca\") pod \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\" (UID: \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\") "
Nov 02 07:39:34.364855 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.364833    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-kube-api-access-rc472\" (UniqueName: \"kubernetes.io/projected/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-kube-api-access-rc472\") pod \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\" (UID: \"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5\") "
Nov 02 07:39:34.390462 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.367369    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:39:34.463312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.463231    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:34.470611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.470582    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-kube-api-access-rc472" pod "cc9f27f0-ab89-41d1-bb4a-0f620923b4d5" (UID: "cc9f27f0-ab89-41d1-bb4a-0f620923b4d5"). InnerVolumeSpecName "kube-api-access-rc472". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:34.509911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.478138    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-xks-ca" pod "cc9f27f0-ab89-41d1-bb4a-0f620923b4d5" (UID: "cc9f27f0-ab89-41d1-bb4a-0f620923b4d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:39:34.627912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.627486    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc472\" (UniqueName: \"kubernetes.io/projected/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-kube-api-access-rc472\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:34.627912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:34.627521    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:35.036594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:35.036553    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 07:39:35.037043 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:35.036625    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 07:39:35.234796 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:35.234761    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"
Nov 02 07:39:35.235020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:35.234793    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch" event={"ID":"cc9f27f0-ab89-41d1-bb4a-0f620923b4d5","Type":"ContainerDied","Data":"20604d18b509a45034516013d34332333fddb7464c5f9700ca35fd14c73b93d3"}
Nov 02 07:39:35.235020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:35.234849    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20604d18b509a45034516013d34332333fddb7464c5f9700ca35fd14c73b93d3"
Nov 02 07:39:36.036829 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.036789    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 07:39:36.037247 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.036854    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 07:39:36.218156 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.218120    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"]
Nov 02 07:39:36.218693 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.218679    4898 cpu_manager.go:414] "containerMap: removing container" podUID="cc9f27f0-ab89-41d1-bb4a-0f620923b4d5" containerName="revoke-keys"
Nov 02 07:39:36.218735 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.218694    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc9f27f0-ab89-41d1-bb4a-0f620923b4d5" containerName="revoke-keys"
Nov 02 07:39:36.218814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.218789    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc9f27f0-ab89-41d1-bb4a-0f620923b4d5" containerName="revoke-keys"
Nov 02 07:39:36.222550 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.222532    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:36.237558 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.237534    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:39:36.245487 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.245459    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6556b310-p7mjd\" (UID: \"69a67f57-f503-4945-81e9-a4d9feadb29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:36.245587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.245516    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6556b310-p7mjd\" (UID: \"69a67f57-f503-4945-81e9-a4d9feadb29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:36.245823 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.245798    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w6g7\" (UniqueName: \"kubernetes.io/projected/69a67f57-f503-4945-81e9-a4d9feadb29a-kube-api-access-2w6g7\") pod \"maas-api-revoke-sub-6556b310-p7mjd\" (UID: \"69a67f57-f503-4945-81e9-a4d9feadb29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:36.249566 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.249543    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:39:36.249839 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.249819    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:39:36.250187 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.250164    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:39:36.269249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.269224    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"]
Nov 02 07:39:36.347290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.347203    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6g7\" (UniqueName: \"kubernetes.io/projected/69a67f57-f503-4945-81e9-a4d9feadb29a-kube-api-access-2w6g7\") pod \"maas-api-revoke-sub-6556b310-p7mjd\" (UID: \"69a67f57-f503-4945-81e9-a4d9feadb29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:36.347290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.347245    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6556b310-p7mjd\" (UID: \"69a67f57-f503-4945-81e9-a4d9feadb29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:36.347290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.347278    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6556b310-p7mjd\" (UID: \"69a67f57-f503-4945-81e9-a4d9feadb29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:36.348151 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.348118    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6556b310-p7mjd\" (UID: \"69a67f57-f503-4945-81e9-a4d9feadb29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:36.349827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.349807    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6556b310-p7mjd\" (UID: \"69a67f57-f503-4945-81e9-a4d9feadb29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:36.382560 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.382530    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6g7\" (UniqueName: \"kubernetes.io/projected/69a67f57-f503-4945-81e9-a4d9feadb29a-kube-api-access-2w6g7\") pod \"maas-api-revoke-sub-6556b310-p7mjd\" (UID: \"69a67f57-f503-4945-81e9-a4d9feadb29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:36.536575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.536522    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:36.883964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:36.883482    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"]
Nov 02 07:39:37.037086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:37.037048    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 07:39:37.037575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:37.037111    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 07:39:37.252663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:37.252626    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd" event={"ID":"69a67f57-f503-4945-81e9-a4d9feadb29a","Type":"ContainerStarted","Data":"4f3a4d277641846da0dd1a698ddbf1d93d348cf502ca1b70ad3bc180808fa26d"}
Nov 02 07:39:38.036859 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:38.036823    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 07:39:38.037050 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:38.036875    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 07:39:38.287861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:38.287760    4898 generic.go:356] "Generic (PLEG): container finished" podID="69a67f57-f503-4945-81e9-a4d9feadb29a" containerID="1392aa99e4c2ad6a9bf94280928f619760dab6716e3dffd0ef674bf3c65511cd" exitCode=0
Nov 02 07:39:38.288302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:38.287852    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd" event={"ID":"69a67f57-f503-4945-81e9-a4d9feadb29a","Type":"ContainerDied","Data":"1392aa99e4c2ad6a9bf94280928f619760dab6716e3dffd0ef674bf3c65511cd"}
Nov 02 07:39:39.037152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:39.037118    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 07:39:39.037326 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:39.037174    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 07:39:39.509435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:39.509097    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:39.608313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:39.608218    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-xks-ca\") pod \"69a67f57-f503-4945-81e9-a4d9feadb29a\" (UID: \"69a67f57-f503-4945-81e9-a4d9feadb29a\") "
Nov 02 07:39:39.608477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:39.608312    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a67f57-f503-4945-81e9-a4d9feadb29a-kube-api-access-2w6g7\" (UniqueName: \"kubernetes.io/projected/69a67f57-f503-4945-81e9-a4d9feadb29a-kube-api-access-2w6g7\") pod \"69a67f57-f503-4945-81e9-a4d9feadb29a\" (UID: \"69a67f57-f503-4945-81e9-a4d9feadb29a\") "
Nov 02 07:39:39.608542 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:39.608502    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-openshift-ca\") pod \"69a67f57-f503-4945-81e9-a4d9feadb29a\" (UID: \"69a67f57-f503-4945-81e9-a4d9feadb29a\") "
Nov 02 07:39:39.609698 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:39.609665    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-openshift-ca" pod "69a67f57-f503-4945-81e9-a4d9feadb29a" (UID: "69a67f57-f503-4945-81e9-a4d9feadb29a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:39.611261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:39.611232    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-xks-ca" pod "69a67f57-f503-4945-81e9-a4d9feadb29a" (UID: "69a67f57-f503-4945-81e9-a4d9feadb29a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:39:39.611934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:39.611876    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a67f57-f503-4945-81e9-a4d9feadb29a-kube-api-access-2w6g7" pod "69a67f57-f503-4945-81e9-a4d9feadb29a" (UID: "69a67f57-f503-4945-81e9-a4d9feadb29a"). InnerVolumeSpecName "kube-api-access-2w6g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:39.710393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:39.710350    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:39.710393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:39.710392    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a67f57-f503-4945-81e9-a4d9feadb29a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:39.710606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:39.710409    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w6g7\" (UniqueName: \"kubernetes.io/projected/69a67f57-f503-4945-81e9-a4d9feadb29a-kube-api-access-2w6g7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:40.315864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:40.315819    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd" event={"ID":"69a67f57-f503-4945-81e9-a4d9feadb29a","Type":"ContainerDied","Data":"4f3a4d277641846da0dd1a698ddbf1d93d348cf502ca1b70ad3bc180808fa26d"}
Nov 02 07:39:40.316061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:40.315937    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f3a4d277641846da0dd1a698ddbf1d93d348cf502ca1b70ad3bc180808fa26d"
Nov 02 07:39:40.316061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:40.316049    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"
Nov 02 07:39:40.322205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:40.322178    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:40.323592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:40.323568    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:40.328316 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:40.328291    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:39:41.999772 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:41.999735    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"]
Nov 02 07:39:42.000630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.000606    4898 cpu_manager.go:414] "containerMap: removing container" podUID="69a67f57-f503-4945-81e9-a4d9feadb29a" containerName="revoke-keys"
Nov 02 07:39:42.000684 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.000634    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a67f57-f503-4945-81e9-a4d9feadb29a" containerName="revoke-keys"
Nov 02 07:39:42.000816 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.000803    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a67f57-f503-4945-81e9-a4d9feadb29a" containerName="revoke-keys"
Nov 02 07:39:42.005162 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.005142    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:42.021836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.021808    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"]
Nov 02 07:39:42.029731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.029090    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:39:42.029731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.029195    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:39:42.029731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.029232    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:39:42.029731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.029237    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:39:42.035700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.035674    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f169cbee-clgtz\" (UID: \"737594a7-28b1-490a-ad31-7e7ffbf30459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:42.035789 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.035721    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwmss\" (UniqueName: \"kubernetes.io/projected/737594a7-28b1-490a-ad31-7e7ffbf30459-kube-api-access-qwmss\") pod \"maas-api-revoke-sub-f169cbee-clgtz\" (UID: \"737594a7-28b1-490a-ad31-7e7ffbf30459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:42.035845 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.035822    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f169cbee-clgtz\" (UID: \"737594a7-28b1-490a-ad31-7e7ffbf30459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:42.137096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.137062    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f169cbee-clgtz\" (UID: \"737594a7-28b1-490a-ad31-7e7ffbf30459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:42.137258 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.137111    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwmss\" (UniqueName: \"kubernetes.io/projected/737594a7-28b1-490a-ad31-7e7ffbf30459-kube-api-access-qwmss\") pod \"maas-api-revoke-sub-f169cbee-clgtz\" (UID: \"737594a7-28b1-490a-ad31-7e7ffbf30459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:42.137258 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.137154    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f169cbee-clgtz\" (UID: \"737594a7-28b1-490a-ad31-7e7ffbf30459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:42.137877 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.137849    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f169cbee-clgtz\" (UID: \"737594a7-28b1-490a-ad31-7e7ffbf30459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:42.139466 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.139447    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f169cbee-clgtz\" (UID: \"737594a7-28b1-490a-ad31-7e7ffbf30459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:42.161065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.161037    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwmss\" (UniqueName: \"kubernetes.io/projected/737594a7-28b1-490a-ad31-7e7ffbf30459-kube-api-access-qwmss\") pod \"maas-api-revoke-sub-f169cbee-clgtz\" (UID: \"737594a7-28b1-490a-ad31-7e7ffbf30459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:42.320089 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.319995    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:42.665772 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:39:42.665733    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737594a7_28b1_490a_ad31_7e7ffbf30459.slice/crio-6edbde84c7c753b033fa4d9976975857d5361715ce252ff3487dfed21982ebdd WatchSource:0}: Error finding container 6edbde84c7c753b033fa4d9976975857d5361715ce252ff3487dfed21982ebdd: Status 404 returned error can't find the container with id 6edbde84c7c753b033fa4d9976975857d5361715ce252ff3487dfed21982ebdd
Nov 02 07:39:42.668362 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:42.668335    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"]
Nov 02 07:39:43.424912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:43.415634    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz" event={"ID":"737594a7-28b1-490a-ad31-7e7ffbf30459","Type":"ContainerStarted","Data":"03f61866c92e9a4520ce82ad5d7348bea7f7dad00b120c20508e54e0f278bf47"}
Nov 02 07:39:43.424912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:43.415705    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz" event={"ID":"737594a7-28b1-490a-ad31-7e7ffbf30459","Type":"ContainerStarted","Data":"6edbde84c7c753b033fa4d9976975857d5361715ce252ff3487dfed21982ebdd"}
Nov 02 07:39:43.520911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:43.520059    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz" podStartSLOduration=2.5200391509999998 podStartE2EDuration="2.520039151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:43.496277843 +0000 UTC m=+934.186604834" watchObservedRunningTime="2026-11-02 07:39:43.520039151 +0000 UTC m=+934.210366116"
Nov 02 07:39:44.484853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:44.484332    4898 generic.go:356] "Generic (PLEG): container finished" podID="737594a7-28b1-490a-ad31-7e7ffbf30459" containerID="03f61866c92e9a4520ce82ad5d7348bea7f7dad00b120c20508e54e0f278bf47" exitCode=0
Nov 02 07:39:44.484853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:44.484438    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz" event={"ID":"737594a7-28b1-490a-ad31-7e7ffbf30459","Type":"ContainerDied","Data":"03f61866c92e9a4520ce82ad5d7348bea7f7dad00b120c20508e54e0f278bf47"}
Nov 02 07:39:45.230701 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:39:45.207438    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:39:45.299001 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:45.259964    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:39:45.384936 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:45.371056    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 02 07:39:45.407652 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:45.407594    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 02 07:39:45.821924 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:45.821881    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:45.927422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:45.927388    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-openshift-ca\") pod \"737594a7-28b1-490a-ad31-7e7ffbf30459\" (UID: \"737594a7-28b1-490a-ad31-7e7ffbf30459\") "
Nov 02 07:39:45.927615 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:45.927461    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-xks-ca\") pod \"737594a7-28b1-490a-ad31-7e7ffbf30459\" (UID: \"737594a7-28b1-490a-ad31-7e7ffbf30459\") "
Nov 02 07:39:45.927615 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:45.927564    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737594a7-28b1-490a-ad31-7e7ffbf30459-kube-api-access-qwmss\" (UniqueName: \"kubernetes.io/projected/737594a7-28b1-490a-ad31-7e7ffbf30459-kube-api-access-qwmss\") pod \"737594a7-28b1-490a-ad31-7e7ffbf30459\" (UID: \"737594a7-28b1-490a-ad31-7e7ffbf30459\") "
Nov 02 07:39:45.927834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:45.927806    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-openshift-ca" pod "737594a7-28b1-490a-ad31-7e7ffbf30459" (UID: "737594a7-28b1-490a-ad31-7e7ffbf30459"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:45.928037 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:45.928018    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:45.929754 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:45.929720    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-xks-ca" pod "737594a7-28b1-490a-ad31-7e7ffbf30459" (UID: "737594a7-28b1-490a-ad31-7e7ffbf30459"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:39:45.929837 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:45.929773    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737594a7-28b1-490a-ad31-7e7ffbf30459-kube-api-access-qwmss" pod "737594a7-28b1-490a-ad31-7e7ffbf30459" (UID: "737594a7-28b1-490a-ad31-7e7ffbf30459"). InnerVolumeSpecName "kube-api-access-qwmss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:46.034912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:46.032912    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwmss\" (UniqueName: \"kubernetes.io/projected/737594a7-28b1-490a-ad31-7e7ffbf30459-kube-api-access-qwmss\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:46.034912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:46.032965    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737594a7-28b1-490a-ad31-7e7ffbf30459-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:46.615324 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:46.615287    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz" event={"ID":"737594a7-28b1-490a-ad31-7e7ffbf30459","Type":"ContainerDied","Data":"6edbde84c7c753b033fa4d9976975857d5361715ce252ff3487dfed21982ebdd"}
Nov 02 07:39:46.615497 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:46.615344    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6edbde84c7c753b033fa4d9976975857d5361715ce252ff3487dfed21982ebdd"
Nov 02 07:39:46.615497 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:46.615413    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"
Nov 02 07:39:47.567278 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.567242    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"]
Nov 02 07:39:47.568096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.568073    4898 cpu_manager.go:414] "containerMap: removing container" podUID="737594a7-28b1-490a-ad31-7e7ffbf30459" containerName="revoke-keys"
Nov 02 07:39:47.568180 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.568099    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737594a7-28b1-490a-ad31-7e7ffbf30459" containerName="revoke-keys"
Nov 02 07:39:47.568281 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.568269    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="737594a7-28b1-490a-ad31-7e7ffbf30459" containerName="revoke-keys"
Nov 02 07:39:47.572328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.572305    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:47.589874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.589848    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"]
Nov 02 07:39:47.597199 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.597173    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:39:47.597300 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.597268    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:39:47.599242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.599221    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:39:47.599357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.599282    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:39:47.665345 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.665314    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc51541b-ktltt\" (UID: \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:47.665494 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.665349    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9vsb\" (UniqueName: \"kubernetes.io/projected/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-kube-api-access-g9vsb\") pod \"maas-api-revoke-sub-fc51541b-ktltt\" (UID: \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:47.665494 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.665387    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc51541b-ktltt\" (UID: \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:47.766531 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.766495    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc51541b-ktltt\" (UID: \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:47.766531 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.766534    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vsb\" (UniqueName: \"kubernetes.io/projected/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-kube-api-access-g9vsb\") pod \"maas-api-revoke-sub-fc51541b-ktltt\" (UID: \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:47.766783 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.766561    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc51541b-ktltt\" (UID: \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:47.767244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.767220    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc51541b-ktltt\" (UID: \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:47.769259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.769231    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc51541b-ktltt\" (UID: \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:47.790277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.790252    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vsb\" (UniqueName: \"kubernetes.io/projected/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-kube-api-access-g9vsb\") pod \"maas-api-revoke-sub-fc51541b-ktltt\" (UID: \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:47.888874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:47.888788    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:48.188048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:48.188012    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"]
Nov 02 07:39:48.188199 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:39:48.188059    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f40e3b_4a74_473b_889b_f7d77c2dd65f.slice/crio-fa0712e637e3121b5be07e118f2ba3ecf8e596108c00b954c96a15cdb3621cb7 WatchSource:0}: Error finding container fa0712e637e3121b5be07e118f2ba3ecf8e596108c00b954c96a15cdb3621cb7: Status 404 returned error can't find the container with id fa0712e637e3121b5be07e118f2ba3ecf8e596108c00b954c96a15cdb3621cb7
Nov 02 07:39:48.631904 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:48.631865    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt" event={"ID":"a8f40e3b-4a74-473b-889b-f7d77c2dd65f","Type":"ContainerStarted","Data":"bb46e37522ab362d12554c7543ec5b12f00cfd24c075e48a0410420543e64f24"}
Nov 02 07:39:48.632312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:48.631950    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt" event={"ID":"a8f40e3b-4a74-473b-889b-f7d77c2dd65f","Type":"ContainerStarted","Data":"fa0712e637e3121b5be07e118f2ba3ecf8e596108c00b954c96a15cdb3621cb7"}
Nov 02 07:39:48.687666 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:48.687608    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt" podStartSLOduration=1.687590956 podStartE2EDuration="1.687590956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:48.671371352 +0000 UTC m=+939.361698336" watchObservedRunningTime="2026-11-02 07:39:48.687590956 +0000 UTC m=+939.377917923"
Nov 02 07:39:48.915721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:48.915685    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"]
Nov 02 07:39:48.920135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:48.920112    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:48.940086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:48.940061    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"]
Nov 02 07:39:49.086334 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.086303    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36376945-8bce-4163-8129-c933996144dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0f501c6-g5jkd\" (UID: \"36376945-8bce-4163-8129-c933996144dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:49.086486 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.086339    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shlpt\" (UniqueName: \"kubernetes.io/projected/36376945-8bce-4163-8129-c933996144dc-kube-api-access-shlpt\") pod \"maas-api-revoke-sub-b0f501c6-g5jkd\" (UID: \"36376945-8bce-4163-8129-c933996144dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:49.086694 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.086666    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36376945-8bce-4163-8129-c933996144dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0f501c6-g5jkd\" (UID: \"36376945-8bce-4163-8129-c933996144dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:49.120746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.120712    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"]
Nov 02 07:39:49.124809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.124770    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.153611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.153579    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"]
Nov 02 07:39:49.187964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.187867    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36376945-8bce-4163-8129-c933996144dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0f501c6-g5jkd\" (UID: \"36376945-8bce-4163-8129-c933996144dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:49.188227 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.188199    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36376945-8bce-4163-8129-c933996144dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0f501c6-g5jkd\" (UID: \"36376945-8bce-4163-8129-c933996144dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:49.188346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.188241    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shlpt\" (UniqueName: \"kubernetes.io/projected/36376945-8bce-4163-8129-c933996144dc-kube-api-access-shlpt\") pod \"maas-api-revoke-sub-b0f501c6-g5jkd\" (UID: \"36376945-8bce-4163-8129-c933996144dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:49.189439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.189417    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36376945-8bce-4163-8129-c933996144dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0f501c6-g5jkd\" (UID: \"36376945-8bce-4163-8129-c933996144dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:49.190539 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.190519    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36376945-8bce-4163-8129-c933996144dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0f501c6-g5jkd\" (UID: \"36376945-8bce-4163-8129-c933996144dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:49.211997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.211971    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shlpt\" (UniqueName: \"kubernetes.io/projected/36376945-8bce-4163-8129-c933996144dc-kube-api-access-shlpt\") pod \"maas-api-revoke-sub-b0f501c6-g5jkd\" (UID: \"36376945-8bce-4163-8129-c933996144dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:49.235184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.235158    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:49.291273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.289787    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4791d4-db06-4c5d-8423-745556f551a3-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.291273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.289850    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.291273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.289959    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snthz\" (UniqueName: \"kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-kube-api-access-snthz\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.291273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.290024    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4791d4-db06-4c5d-8423-745556f551a3-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.291273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.290064    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.291273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.290115    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.291273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.290171    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.291273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.290218    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.291273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.290280    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.391598 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.391564    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.391751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.391664    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4791d4-db06-4c5d-8423-745556f551a3-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.391751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.391700    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.391751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.391725    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snthz\" (UniqueName: \"kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-kube-api-access-snthz\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.391989 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.391757    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4791d4-db06-4c5d-8423-745556f551a3-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.391989 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.391795    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.391989 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.391838    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.391989 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.391898    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.391989 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.391942    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.392309 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.392285    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-certs\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.392557 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.392503    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.392557 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.392548    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-data\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.392739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.392579    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-credential-socket\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.392739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.392655    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4791d4-db06-4c5d-8423-745556f551a3-istiod-ca-cert\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.394071 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.394050    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-envoy\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.394318 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.394301    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4791d4-db06-4c5d-8423-745556f551a3-istio-podinfo\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.414972 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.414935    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snthz\" (UniqueName: \"kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-kube-api-access-snthz\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.415056 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.414947    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-istio-token\") pod \"e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") " pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.455111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.455045    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:49.539004 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:39:49.538962    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36376945_8bce_4163_8129_c933996144dc.slice/crio-7d797e9e13280433f3e555513e91b7c9f23ef2150ff1eff9a5a5d488ad0eb4c0 WatchSource:0}: Error finding container 7d797e9e13280433f3e555513e91b7c9f23ef2150ff1eff9a5a5d488ad0eb4c0: Status 404 returned error can't find the container with id 7d797e9e13280433f3e555513e91b7c9f23ef2150ff1eff9a5a5d488ad0eb4c0
Nov 02 07:39:49.539173 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.539148    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"]
Nov 02 07:39:49.640725 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.640691    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd" event={"ID":"36376945-8bce-4163-8129-c933996144dc","Type":"ContainerStarted","Data":"7d797e9e13280433f3e555513e91b7c9f23ef2150ff1eff9a5a5d488ad0eb4c0"}
Nov 02 07:39:49.642440 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.642415    4898 generic.go:356] "Generic (PLEG): container finished" podID="a8f40e3b-4a74-473b-889b-f7d77c2dd65f" containerID="bb46e37522ab362d12554c7543ec5b12f00cfd24c075e48a0410420543e64f24" exitCode=0
Nov 02 07:39:49.642551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.642464    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt" event={"ID":"a8f40e3b-4a74-473b-889b-f7d77c2dd65f","Type":"ContainerDied","Data":"bb46e37522ab362d12554c7543ec5b12f00cfd24c075e48a0410420543e64f24"}
Nov 02 07:39:49.768643 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.768610    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"]
Nov 02 07:39:49.770429 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:39:49.770386    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4791d4_db06_4c5d_8423_745556f551a3.slice/crio-e91ecaa402f0e37661683cb4408d94e49c57fa6e0a5021a69c5880f787c82dbb WatchSource:0}: Error finding container e91ecaa402f0e37661683cb4408d94e49c57fa6e0a5021a69c5880f787c82dbb: Status 404 returned error can't find the container with id e91ecaa402f0e37661683cb4408d94e49c57fa6e0a5021a69c5880f787c82dbb
Nov 02 07:39:49.774165 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.774120    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:49.774275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.774213    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:49.774275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:49.774256    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:39:50.688913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:50.688719    4898 generic.go:356] "Generic (PLEG): container finished" podID="36376945-8bce-4163-8129-c933996144dc" containerID="625a055bce0408c76cde64bbd5bc2eb310edbfbe6b75b975668a94b167ce62e7" exitCode=0
Nov 02 07:39:50.689379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:50.689064    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd" event={"ID":"36376945-8bce-4163-8129-c933996144dc","Type":"ContainerDied","Data":"625a055bce0408c76cde64bbd5bc2eb310edbfbe6b75b975668a94b167ce62e7"}
Nov 02 07:39:50.692907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:50.692766    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp" event={"ID":"3f4791d4-db06-4c5d-8423-745556f551a3","Type":"ContainerStarted","Data":"ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be"}
Nov 02 07:39:50.692907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:50.692837    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp" event={"ID":"3f4791d4-db06-4c5d-8423-745556f551a3","Type":"ContainerStarted","Data":"e91ecaa402f0e37661683cb4408d94e49c57fa6e0a5021a69c5880f787c82dbb"}
Nov 02 07:39:50.805784 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:50.805720    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp" podStartSLOduration=1.8056991340000002 podStartE2EDuration="1.805699134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:50.788457854 +0000 UTC m=+941.478784828" watchObservedRunningTime="2026-11-02 07:39:50.805699134 +0000 UTC m=+941.496026099"
Nov 02 07:39:50.983253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:50.983212    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:51.111486 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.111436    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-xks-ca\") pod \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\" (UID: \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\") "
Nov 02 07:39:51.111637 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.111535    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-kube-api-access-g9vsb\" (UniqueName: \"kubernetes.io/projected/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-kube-api-access-g9vsb\") pod \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\" (UID: \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\") "
Nov 02 07:39:51.111761 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.111744    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-openshift-ca\") pod \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\" (UID: \"a8f40e3b-4a74-473b-889b-f7d77c2dd65f\") "
Nov 02 07:39:51.112177 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.112136    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-openshift-ca" pod "a8f40e3b-4a74-473b-889b-f7d77c2dd65f" (UID: "a8f40e3b-4a74-473b-889b-f7d77c2dd65f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:51.112422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.112382    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:51.113731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.113697    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-xks-ca" pod "a8f40e3b-4a74-473b-889b-f7d77c2dd65f" (UID: "a8f40e3b-4a74-473b-889b-f7d77c2dd65f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:39:51.113829 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.113802    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-kube-api-access-g9vsb" pod "a8f40e3b-4a74-473b-889b-f7d77c2dd65f" (UID: "a8f40e3b-4a74-473b-889b-f7d77c2dd65f"). InnerVolumeSpecName "kube-api-access-g9vsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:51.213674 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.213633    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:51.213674 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.213668    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9vsb\" (UniqueName: \"kubernetes.io/projected/a8f40e3b-4a74-473b-889b-f7d77c2dd65f-kube-api-access-g9vsb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:51.469034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.468944    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:51.486205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.481805    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 07:39:51.486205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.481860    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp" podUID="3f4791d4-db06-4c5d-8423-745556f551a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 07:39:51.700647 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.700615    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"
Nov 02 07:39:51.701152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.700654    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt" event={"ID":"a8f40e3b-4a74-473b-889b-f7d77c2dd65f","Type":"ContainerDied","Data":"fa0712e637e3121b5be07e118f2ba3ecf8e596108c00b954c96a15cdb3621cb7"}
Nov 02 07:39:51.701152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.700695    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa0712e637e3121b5be07e118f2ba3ecf8e596108c00b954c96a15cdb3621cb7"
Nov 02 07:39:51.916452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:51.916430    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:52.024701 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.024611    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36376945-8bce-4163-8129-c933996144dc-kube-api-access-shlpt\" (UniqueName: \"kubernetes.io/projected/36376945-8bce-4163-8129-c933996144dc-kube-api-access-shlpt\") pod \"36376945-8bce-4163-8129-c933996144dc\" (UID: \"36376945-8bce-4163-8129-c933996144dc\") "
Nov 02 07:39:52.024860 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.024754    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36376945-8bce-4163-8129-c933996144dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36376945-8bce-4163-8129-c933996144dc-maas-api-openshift-ca\") pod \"36376945-8bce-4163-8129-c933996144dc\" (UID: \"36376945-8bce-4163-8129-c933996144dc\") "
Nov 02 07:39:52.024956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.024878    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36376945-8bce-4163-8129-c933996144dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36376945-8bce-4163-8129-c933996144dc-maas-api-xks-ca\") pod \"36376945-8bce-4163-8129-c933996144dc\" (UID: \"36376945-8bce-4163-8129-c933996144dc\") "
Nov 02 07:39:52.025270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.025239    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36376945-8bce-4163-8129-c933996144dc-maas-api-openshift-ca" pod "36376945-8bce-4163-8129-c933996144dc" (UID: "36376945-8bce-4163-8129-c933996144dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:52.025525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.025506    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36376945-8bce-4163-8129-c933996144dc-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:52.026943 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.026920    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36376945-8bce-4163-8129-c933996144dc-kube-api-access-shlpt" pod "36376945-8bce-4163-8129-c933996144dc" (UID: "36376945-8bce-4163-8129-c933996144dc"). InnerVolumeSpecName "kube-api-access-shlpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:52.027048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.027028    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36376945-8bce-4163-8129-c933996144dc-maas-api-xks-ca" pod "36376945-8bce-4163-8129-c933996144dc" (UID: "36376945-8bce-4163-8129-c933996144dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:39:52.127264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.127235    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36376945-8bce-4163-8129-c933996144dc-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:52.127264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.127262    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shlpt\" (UniqueName: \"kubernetes.io/projected/36376945-8bce-4163-8129-c933996144dc-kube-api-access-shlpt\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:52.419841 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.419806    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"]
Nov 02 07:39:52.420446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.420427    4898 cpu_manager.go:414] "containerMap: removing container" podUID="36376945-8bce-4163-8129-c933996144dc" containerName="revoke-keys"
Nov 02 07:39:52.420446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.420443    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36376945-8bce-4163-8129-c933996144dc" containerName="revoke-keys"
Nov 02 07:39:52.420637 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.420456    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a8f40e3b-4a74-473b-889b-f7d77c2dd65f" containerName="revoke-keys"
Nov 02 07:39:52.420637 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.420461    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f40e3b-4a74-473b-889b-f7d77c2dd65f" containerName="revoke-keys"
Nov 02 07:39:52.420637 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.420567    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="36376945-8bce-4163-8129-c933996144dc" containerName="revoke-keys"
Nov 02 07:39:52.420637 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.420576    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f40e3b-4a74-473b-889b-f7d77c2dd65f" containerName="revoke-keys"
Nov 02 07:39:52.424285 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.424264    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:52.450452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.450421    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"]
Nov 02 07:39:52.483923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.482410    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 07:39:52.483923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.482470    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp" podUID="3f4791d4-db06-4c5d-8423-745556f551a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 07:39:52.532602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.532565    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9034bc-k8xq2\" (UID: \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:52.532733 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.532713    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9034bc-k8xq2\" (UID: \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:52.532790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.532781    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjq7g\" (UniqueName: \"kubernetes.io/projected/42c627f6-410b-4f79-b096-f74f6ac6dc3a-kube-api-access-pjq7g\") pod \"maas-api-revoke-sub-ac9034bc-k8xq2\" (UID: \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:52.633872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.633835    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9034bc-k8xq2\" (UID: \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:52.634046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.633920    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjq7g\" (UniqueName: \"kubernetes.io/projected/42c627f6-410b-4f79-b096-f74f6ac6dc3a-kube-api-access-pjq7g\") pod \"maas-api-revoke-sub-ac9034bc-k8xq2\" (UID: \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:52.634099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.634054    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9034bc-k8xq2\" (UID: \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:52.634611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.634586    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9034bc-k8xq2\" (UID: \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:52.636408 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.636380    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9034bc-k8xq2\" (UID: \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:52.674802 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.674719    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjq7g\" (UniqueName: \"kubernetes.io/projected/42c627f6-410b-4f79-b096-f74f6ac6dc3a-kube-api-access-pjq7g\") pod \"maas-api-revoke-sub-ac9034bc-k8xq2\" (UID: \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:52.713265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.713222    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd" event={"ID":"36376945-8bce-4163-8129-c933996144dc","Type":"ContainerDied","Data":"7d797e9e13280433f3e555513e91b7c9f23ef2150ff1eff9a5a5d488ad0eb4c0"}
Nov 02 07:39:52.713657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.713292    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d797e9e13280433f3e555513e91b7c9f23ef2150ff1eff9a5a5d488ad0eb4c0"
Nov 02 07:39:52.713657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.713239    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"
Nov 02 07:39:52.740657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:52.740625    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:53.105554 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:53.105511    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"]
Nov 02 07:39:53.470019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:53.469983    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 07:39:53.470312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:53.470283    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp" podUID="3f4791d4-db06-4c5d-8423-745556f551a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 07:39:53.722217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:53.722116    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2" event={"ID":"42c627f6-410b-4f79-b096-f74f6ac6dc3a","Type":"ContainerStarted","Data":"c171a739726425d209be672699f0a94c4488a8df35bfec18b8b3806f5b49aa29"}
Nov 02 07:39:53.722217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:53.722175    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2" event={"ID":"42c627f6-410b-4f79-b096-f74f6ac6dc3a","Type":"ContainerStarted","Data":"1f22440aa2a2df62c02cb2f2d6018155b25a682809392f208b876308a3bddce5"}
Nov 02 07:39:54.470932 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:54.469988    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 07:39:54.470932 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:54.470049    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp" podUID="3f4791d4-db06-4c5d-8423-745556f551a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 07:39:54.731011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:54.730972    4898 generic.go:356] "Generic (PLEG): container finished" podID="42c627f6-410b-4f79-b096-f74f6ac6dc3a" containerID="c171a739726425d209be672699f0a94c4488a8df35bfec18b8b3806f5b49aa29" exitCode=0
Nov 02 07:39:54.731408 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:54.731041    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2" event={"ID":"42c627f6-410b-4f79-b096-f74f6ac6dc3a","Type":"ContainerDied","Data":"c171a739726425d209be672699f0a94c4488a8df35bfec18b8b3806f5b49aa29"}
Nov 02 07:39:55.476913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:55.460243    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 07:39:55.476913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:55.460315    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp" podUID="3f4791d4-db06-4c5d-8423-745556f551a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 07:39:55.939030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:55.939003    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:56.012986 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.012866    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c627f6-410b-4f79-b096-f74f6ac6dc3a-kube-api-access-pjq7g\" (UniqueName: \"kubernetes.io/projected/42c627f6-410b-4f79-b096-f74f6ac6dc3a-kube-api-access-pjq7g\") pod \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\" (UID: \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\") "
Nov 02 07:39:56.013154 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.013067    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-openshift-ca\") pod \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\" (UID: \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\") "
Nov 02 07:39:56.013154 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.013117    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-xks-ca\") pod \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\" (UID: \"42c627f6-410b-4f79-b096-f74f6ac6dc3a\") "
Nov 02 07:39:56.013452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.013418    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-openshift-ca" pod "42c627f6-410b-4f79-b096-f74f6ac6dc3a" (UID: "42c627f6-410b-4f79-b096-f74f6ac6dc3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:39:56.013683 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.013659    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:56.015211 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.015187    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c627f6-410b-4f79-b096-f74f6ac6dc3a-kube-api-access-pjq7g" pod "42c627f6-410b-4f79-b096-f74f6ac6dc3a" (UID: "42c627f6-410b-4f79-b096-f74f6ac6dc3a"). InnerVolumeSpecName "kube-api-access-pjq7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:39:56.015401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.015373    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-xks-ca" pod "42c627f6-410b-4f79-b096-f74f6ac6dc3a" (UID: "42c627f6-410b-4f79-b096-f74f6ac6dc3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:39:56.115104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.115068    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42c627f6-410b-4f79-b096-f74f6ac6dc3a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:56.115104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.115097    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjq7g\" (UniqueName: \"kubernetes.io/projected/42c627f6-410b-4f79-b096-f74f6ac6dc3a-kube-api-access-pjq7g\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:39:56.750384 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.750340    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2" event={"ID":"42c627f6-410b-4f79-b096-f74f6ac6dc3a","Type":"ContainerDied","Data":"1f22440aa2a2df62c02cb2f2d6018155b25a682809392f208b876308a3bddce5"}
Nov 02 07:39:56.750600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.750401    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f22440aa2a2df62c02cb2f2d6018155b25a682809392f208b876308a3bddce5"
Nov 02 07:39:56.750600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.750355    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"
Nov 02 07:39:56.849517 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.849479    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:56.853780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.850047    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:56.853780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:56.853254    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:39:57.407858 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.407825    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"]
Nov 02 07:39:57.408452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.408434    4898 cpu_manager.go:414] "containerMap: removing container" podUID="42c627f6-410b-4f79-b096-f74f6ac6dc3a" containerName="revoke-keys"
Nov 02 07:39:57.408452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.408451    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c627f6-410b-4f79-b096-f74f6ac6dc3a" containerName="revoke-keys"
Nov 02 07:39:57.408590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.408542    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c627f6-410b-4f79-b096-f74f6ac6dc3a" containerName="revoke-keys"
Nov 02 07:39:57.412021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.411998    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.440995 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.440929    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-354660\"/\"distinct-2-w3-354660-kserve-self-signed-certs\""
Nov 02 07:39:57.449130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.449098    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"]
Nov 02 07:39:57.531152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.531121    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tls-certs\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.531322 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.531184    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-dshm\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.531322 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.531306    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tmp-dir\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.531418 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.531338    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blfhw\" (UniqueName: \"kubernetes.io/projected/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-kube-api-access-blfhw\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.531467 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.531432    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-home\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.531513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.531486    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-model-cache\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.632692 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.632650    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tls-certs\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.632846 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.632742    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-dshm\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.632846 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.632823    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tmp-dir\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.633004 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.632846    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blfhw\" (UniqueName: \"kubernetes.io/projected/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-kube-api-access-blfhw\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.633004 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.632881    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-home\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.633004 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.632958    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-model-cache\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.633293 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.633265    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tmp-dir\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.633383 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.633324    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-model-cache\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.633383 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.633354    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-home\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.635192 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.635168    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-dshm\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.635365 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.635347    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tls-certs\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.656299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.656270    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blfhw\" (UniqueName: \"kubernetes.io/projected/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-kube-api-access-blfhw\") pod \"distinct-2-w3-354660-kserve-86c6cff557-xnrtc\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") " pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:57.738684 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:57.737161    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:58.064679 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:39:58.064644    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71cd64dc_bdb6_4d4b_ac78_bf67f4795702.slice/crio-e4a4bc3d0f96db0675bae2ec5d4effcf954e1a5d507070bd5678339bc948a09b WatchSource:0}: Error finding container e4a4bc3d0f96db0675bae2ec5d4effcf954e1a5d507070bd5678339bc948a09b: Status 404 returned error can't find the container with id e4a4bc3d0f96db0675bae2ec5d4effcf954e1a5d507070bd5678339bc948a09b
Nov 02 07:39:58.066914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:58.066875    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"]
Nov 02 07:39:58.774452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:58.774416    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" event={"ID":"71cd64dc-bdb6-4d4b-ac78-bf67f4795702","Type":"ContainerStarted","Data":"8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c"}
Nov 02 07:39:58.774908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:58.774466    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" event={"ID":"71cd64dc-bdb6-4d4b-ac78-bf67f4795702","Type":"ContainerStarted","Data":"e4a4bc3d0f96db0675bae2ec5d4effcf954e1a5d507070bd5678339bc948a09b"}
Nov 02 07:39:58.820665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:58.820602    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podStartSLOduration=1.820583246 podStartE2EDuration="1.820583246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:39:58.817361827 +0000 UTC m=+949.507688814" watchObservedRunningTime="2026-11-02 07:39:58.820583246 +0000 UTC m=+949.510910211"
Nov 02 07:39:59.502912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.498453    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"]
Nov 02 07:39:59.609622 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.608816    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"]
Nov 02 07:39:59.609622 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.609013    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.640315 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.640285    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 07:39:59.640592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.640564    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-ctb6j\""
Nov 02 07:39:59.640693 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.640576    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0a310b1a5611\""
Nov 02 07:39:59.640693 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.640663    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 07:39:59.640810 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.640729    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 07:39:59.640998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.640978    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 07:39:59.702605 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.702561    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65825\" (UniqueName: \"kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-kube-api-access-65825\") pod \"payload-processing-external-model-66bbdd4b74-mfg7f\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.702814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.702642    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-plugins-config-volume\") pod \"payload-processing-external-model-66bbdd4b74-mfg7f\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.702814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.702696    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-routing-overlay-volume\") pod \"payload-processing-external-model-66bbdd4b74-mfg7f\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.702814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.702799    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-provider-credentials\") pod \"payload-processing-external-model-66bbdd4b74-mfg7f\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.746911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.739997    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:39:59.807958 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.807846    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-plugins-config-volume\") pod \"payload-processing-external-model-66bbdd4b74-mfg7f\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.808417 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.808096    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-routing-overlay-volume\") pod \"payload-processing-external-model-66bbdd4b74-mfg7f\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.808949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.808924    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-routing-overlay-volume\") pod \"payload-processing-external-model-66bbdd4b74-mfg7f\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.823230 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.823201    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-provider-credentials\") pod \"payload-processing-external-model-66bbdd4b74-mfg7f\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.825029 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.825004    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65825\" (UniqueName: \"kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-kube-api-access-65825\") pod \"payload-processing-external-model-66bbdd4b74-mfg7f\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.825562 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.823543    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-plugins-config-volume\") pod \"payload-processing-external-model-66bbdd4b74-mfg7f\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.842504 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.842476    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-provider-credentials\") pod \"payload-processing-external-model-66bbdd4b74-mfg7f\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.867808 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.867748    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65825\" (UniqueName: \"kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-kube-api-access-65825\") pod \"payload-processing-external-model-66bbdd4b74-mfg7f\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") " pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:39:59.963916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:39:59.961651    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:40:00.674962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:00.221032    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"]
Nov 02 07:40:00.770806 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:00.769300    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:00.783360 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:00.783285    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e04da48_887d_47f1_960c_ae268b1c7586.slice/crio-cdc89798b50a13aee94e105b89f19c266a382968d7b830cd2b25d97a8d7fb561 WatchSource:0}: Error finding container cdc89798b50a13aee94e105b89f19c266a382968d7b830cd2b25d97a8d7fb561: Status 404 returned error can't find the container with id cdc89798b50a13aee94e105b89f19c266a382968d7b830cd2b25d97a8d7fb561
Nov 02 07:40:00.814939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:00.798080    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:40:00.814939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:00.798862    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:40:00.814939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:00.798941    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:40:00.814939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:00.799008    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:40:00.814939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:00.801413    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"]
Nov 02 07:40:00.896968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:00.880339    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dec9e46e-l44qw\" (UID: \"14c9b913-34dc-4904-a36b-11651b8ef06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:00.896968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:00.880422    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dec9e46e-l44qw\" (UID: \"14c9b913-34dc-4904-a36b-11651b8ef06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:00.896968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:00.880465    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ctc\" (UniqueName: \"kubernetes.io/projected/14c9b913-34dc-4904-a36b-11651b8ef06e-kube-api-access-42ctc\") pod \"maas-api-revoke-sub-dec9e46e-l44qw\" (UID: \"14c9b913-34dc-4904-a36b-11651b8ef06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:00.975087 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:00.974933    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"]
Nov 02 07:40:01.043916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.001298    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ctc\" (UniqueName: \"kubernetes.io/projected/14c9b913-34dc-4904-a36b-11651b8ef06e-kube-api-access-42ctc\") pod \"maas-api-revoke-sub-dec9e46e-l44qw\" (UID: \"14c9b913-34dc-4904-a36b-11651b8ef06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:01.043916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.001571    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dec9e46e-l44qw\" (UID: \"14c9b913-34dc-4904-a36b-11651b8ef06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:01.043916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.001666    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dec9e46e-l44qw\" (UID: \"14c9b913-34dc-4904-a36b-11651b8ef06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:01.050912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.046092    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dec9e46e-l44qw\" (UID: \"14c9b913-34dc-4904-a36b-11651b8ef06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:01.104421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.104014    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ctc\" (UniqueName: \"kubernetes.io/projected/14c9b913-34dc-4904-a36b-11651b8ef06e-kube-api-access-42ctc\") pod \"maas-api-revoke-sub-dec9e46e-l44qw\" (UID: \"14c9b913-34dc-4904-a36b-11651b8ef06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:01.117034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.112353    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dec9e46e-l44qw\" (UID: \"14c9b913-34dc-4904-a36b-11651b8ef06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:01.194910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.194785    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:01.201486 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.201440    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f" event={"ID":"7e04da48-887d-47f1-960c-ae268b1c7586","Type":"ContainerStarted","Data":"cdc89798b50a13aee94e105b89f19c266a382968d7b830cd2b25d97a8d7fb561"}
Nov 02 07:40:01.521196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.517720    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:01.521196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.518114    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 02 07:40:01.554013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.542986    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:01.554013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.543324    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 07:40:01.556216 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.555447    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:01.895955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.890435    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:40:01.901103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:01.900987    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:02.364330 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.341710    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Nov 02 07:40:02.393162 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.385953    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 02 07:40:02.461758 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.445910    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f" event={"ID":"7e04da48-887d-47f1-960c-ae268b1c7586","Type":"ContainerStarted","Data":"c444742c42adfc88b5e68224b1ff8508bf1c188374738f78ccebfb6428607af0"}
Nov 02 07:40:02.471205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.469131    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:40:02.696919 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.673858    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 02 07:40:02.700914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.683038    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:40:02.709913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.683072    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 07:40:02.761921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.699318    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:40:02.761921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.710236    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:40:02.761921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.710284    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:40:02.821913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.808960    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f" podStartSLOduration=3.808937534 podStartE2EDuration="3.808937534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:02.670874363 +0000 UTC m=+953.361201327" watchObservedRunningTime="2026-11-02 07:40:02.808937534 +0000 UTC m=+953.499264500"
Nov 02 07:40:02.912917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.906561    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:02.912917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:02.908091    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:40:03.101776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:03.101689    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"]
Nov 02 07:40:03.153909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:03.145147    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:40:03.162912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:03.154310    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:40:03.347632 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:03.341836    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:03.363942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:03.354387    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:40:03.413368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:03.410341    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:40:03.456511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:03.456474    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw" event={"ID":"14c9b913-34dc-4904-a36b-11651b8ef06e","Type":"ContainerStarted","Data":"dff0901bfe0804c0df59742bf31b0436a3cc72fed528a13c2061b52721eaadd8"}
Nov 02 07:40:04.465543 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:04.465461    4898 generic.go:356] "Generic (PLEG): container finished" podID="14c9b913-34dc-4904-a36b-11651b8ef06e" containerID="9fe9f6fff63144d2d309a40717bf918f93ce2abb44c40d7255268a7870b370de" exitCode=0
Nov 02 07:40:04.466066 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:04.465540    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw" event={"ID":"14c9b913-34dc-4904-a36b-11651b8ef06e","Type":"ContainerDied","Data":"9fe9f6fff63144d2d309a40717bf918f93ce2abb44c40d7255268a7870b370de"}
Nov 02 07:40:05.506488 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.506418    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"]
Nov 02 07:40:05.512651 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.512616    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:05.527282 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.526363    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"]
Nov 02 07:40:05.557177 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.554648    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d33fa69-vvhsj\" (UID: \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:05.557177 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.554862    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49rs\" (UniqueName: \"kubernetes.io/projected/1cb4ced7-494b-4050-8a5a-8a604bf056c4-kube-api-access-j49rs\") pod \"maas-api-revoke-sub-4d33fa69-vvhsj\" (UID: \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:05.557177 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.554970    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d33fa69-vvhsj\" (UID: \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:05.656092 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.656057    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49rs\" (UniqueName: \"kubernetes.io/projected/1cb4ced7-494b-4050-8a5a-8a604bf056c4-kube-api-access-j49rs\") pod \"maas-api-revoke-sub-4d33fa69-vvhsj\" (UID: \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:05.656268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.656105    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d33fa69-vvhsj\" (UID: \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:05.656268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.656212    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d33fa69-vvhsj\" (UID: \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:05.657043 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.657015    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d33fa69-vvhsj\" (UID: \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:05.658672 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.658644    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d33fa69-vvhsj\" (UID: \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:05.679343 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.679320    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49rs\" (UniqueName: \"kubernetes.io/projected/1cb4ced7-494b-4050-8a5a-8a604bf056c4-kube-api-access-j49rs\") pod \"maas-api-revoke-sub-4d33fa69-vvhsj\" (UID: \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:05.717068 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.717039    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:05.757295 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.757206    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-openshift-ca\") pod \"14c9b913-34dc-4904-a36b-11651b8ef06e\" (UID: \"14c9b913-34dc-4904-a36b-11651b8ef06e\") "
Nov 02 07:40:05.757469 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.757447    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14c9b913-34dc-4904-a36b-11651b8ef06e-kube-api-access-42ctc\" (UniqueName: \"kubernetes.io/projected/14c9b913-34dc-4904-a36b-11651b8ef06e-kube-api-access-42ctc\") pod \"14c9b913-34dc-4904-a36b-11651b8ef06e\" (UID: \"14c9b913-34dc-4904-a36b-11651b8ef06e\") "
Nov 02 07:40:05.757534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.757487    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-xks-ca\") pod \"14c9b913-34dc-4904-a36b-11651b8ef06e\" (UID: \"14c9b913-34dc-4904-a36b-11651b8ef06e\") "
Nov 02 07:40:05.758075 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.757648    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-openshift-ca" pod "14c9b913-34dc-4904-a36b-11651b8ef06e" (UID: "14c9b913-34dc-4904-a36b-11651b8ef06e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:05.758075 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.758024    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:05.763297 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.763269    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c9b913-34dc-4904-a36b-11651b8ef06e-kube-api-access-42ctc" pod "14c9b913-34dc-4904-a36b-11651b8ef06e" (UID: "14c9b913-34dc-4904-a36b-11651b8ef06e"). InnerVolumeSpecName "kube-api-access-42ctc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:05.764268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.764227    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-xks-ca" pod "14c9b913-34dc-4904-a36b-11651b8ef06e" (UID: "14c9b913-34dc-4904-a36b-11651b8ef06e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:40:05.842373 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.842330    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:05.859497 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.859471    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42ctc\" (UniqueName: \"kubernetes.io/projected/14c9b913-34dc-4904-a36b-11651b8ef06e-kube-api-access-42ctc\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:05.859588 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:05.859501    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14c9b913-34dc-4904-a36b-11651b8ef06e-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:06.146196 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:06.146156    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb4ced7_494b_4050_8a5a_8a604bf056c4.slice/crio-f7c4ebc07586ac90240be544ee5abc33b5beddf34e2b414715b50b2f20fd4f96 WatchSource:0}: Error finding container f7c4ebc07586ac90240be544ee5abc33b5beddf34e2b414715b50b2f20fd4f96: Status 404 returned error can't find the container with id f7c4ebc07586ac90240be544ee5abc33b5beddf34e2b414715b50b2f20fd4f96
Nov 02 07:40:06.147391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:06.147363    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"]
Nov 02 07:40:06.483098 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:06.483061    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj" event={"ID":"1cb4ced7-494b-4050-8a5a-8a604bf056c4","Type":"ContainerStarted","Data":"07e0edf66187f3b2d3402488e0454f60c392fc969f4fcee2ef8ba2cd09e4bf65"}
Nov 02 07:40:06.483236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:06.483131    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj" event={"ID":"1cb4ced7-494b-4050-8a5a-8a604bf056c4","Type":"ContainerStarted","Data":"f7c4ebc07586ac90240be544ee5abc33b5beddf34e2b414715b50b2f20fd4f96"}
Nov 02 07:40:06.484881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:06.484850    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"
Nov 02 07:40:06.484993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:06.484874    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw" event={"ID":"14c9b913-34dc-4904-a36b-11651b8ef06e","Type":"ContainerDied","Data":"dff0901bfe0804c0df59742bf31b0436a3cc72fed528a13c2061b52721eaadd8"}
Nov 02 07:40:06.484993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:06.484931    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dff0901bfe0804c0df59742bf31b0436a3cc72fed528a13c2061b52721eaadd8"
Nov 02 07:40:06.523986 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:06.523941    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj" podStartSLOduration=1.5238794530000002 podStartE2EDuration="1.523879453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:06.523240088 +0000 UTC m=+957.213567053" watchObservedRunningTime="2026-11-02 07:40:06.523879453 +0000 UTC m=+957.214206432"
Nov 02 07:40:07.146264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.146233    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"]
Nov 02 07:40:07.146840 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.146827    4898 cpu_manager.go:414] "containerMap: removing container" podUID="14c9b913-34dc-4904-a36b-11651b8ef06e" containerName="revoke-keys"
Nov 02 07:40:07.146879 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.146841    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14c9b913-34dc-4904-a36b-11651b8ef06e" containerName="revoke-keys"
Nov 02 07:40:07.146971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.146961    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="14c9b913-34dc-4904-a36b-11651b8ef06e" containerName="revoke-keys"
Nov 02 07:40:07.150515 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.150498    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:07.171806 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.171774    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"]
Nov 02 07:40:07.177468 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.177438    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzsjf\" (UniqueName: \"kubernetes.io/projected/da36ae40-a2c8-432a-bc54-6c1097e97b1a-kube-api-access-xzsjf\") pod \"maas-api-revoke-sub-9d19a4f8-g2px7\" (UID: \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:07.177629 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.177499    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d19a4f8-g2px7\" (UID: \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:07.177698 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.177677    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d19a4f8-g2px7\" (UID: \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:07.278451 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.278403    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzsjf\" (UniqueName: \"kubernetes.io/projected/da36ae40-a2c8-432a-bc54-6c1097e97b1a-kube-api-access-xzsjf\") pod \"maas-api-revoke-sub-9d19a4f8-g2px7\" (UID: \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:07.278610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.278479    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d19a4f8-g2px7\" (UID: \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:07.278610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.278558    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d19a4f8-g2px7\" (UID: \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:07.279167 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.279138    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d19a4f8-g2px7\" (UID: \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:07.280789 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.280769    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d19a4f8-g2px7\" (UID: \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:07.303769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.303744    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzsjf\" (UniqueName: \"kubernetes.io/projected/da36ae40-a2c8-432a-bc54-6c1097e97b1a-kube-api-access-xzsjf\") pod \"maas-api-revoke-sub-9d19a4f8-g2px7\" (UID: \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:07.468048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.468014    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:07.497093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.497063    4898 generic.go:356] "Generic (PLEG): container finished" podID="1cb4ced7-494b-4050-8a5a-8a604bf056c4" containerID="07e0edf66187f3b2d3402488e0454f60c392fc969f4fcee2ef8ba2cd09e4bf65" exitCode=0
Nov 02 07:40:07.497219 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.497147    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj" event={"ID":"1cb4ced7-494b-4050-8a5a-8a604bf056c4","Type":"ContainerDied","Data":"07e0edf66187f3b2d3402488e0454f60c392fc969f4fcee2ef8ba2cd09e4bf65"}
Nov 02 07:40:07.737685 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.737586    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:40:07.752040 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.752008    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:40:07.771873 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:07.770152    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda36ae40_a2c8_432a_bc54_6c1097e97b1a.slice/crio-61d14c83eafed65b4a1b30583ff3e79fcc243134ae355b4d5430891a6c50db15 WatchSource:0}: Error finding container 61d14c83eafed65b4a1b30583ff3e79fcc243134ae355b4d5430891a6c50db15: Status 404 returned error can't find the container with id 61d14c83eafed65b4a1b30583ff3e79fcc243134ae355b4d5430891a6c50db15
Nov 02 07:40:07.772545 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:07.772520    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"]
Nov 02 07:40:08.505746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.505713    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7" event={"ID":"da36ae40-a2c8-432a-bc54-6c1097e97b1a","Type":"ContainerStarted","Data":"7a75bb67920e0be74b802d7c0b97efb8e7123bff9b42d96e41329970d8dfd674"}
Nov 02 07:40:08.505955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.505770    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7" event={"ID":"da36ae40-a2c8-432a-bc54-6c1097e97b1a","Type":"ContainerStarted","Data":"61d14c83eafed65b4a1b30583ff3e79fcc243134ae355b4d5430891a6c50db15"}
Nov 02 07:40:08.518232 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.518207    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:40:08.546506 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.546454    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7" podStartSLOduration=1.546441457 podStartE2EDuration="1.546441457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:08.545323266 +0000 UTC m=+959.235650231" watchObservedRunningTime="2026-11-02 07:40:08.546441457 +0000 UTC m=+959.236768421"
Nov 02 07:40:08.678665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.678635    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:08.804401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.804315    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb4ced7-494b-4050-8a5a-8a604bf056c4-kube-api-access-j49rs\" (UniqueName: \"kubernetes.io/projected/1cb4ced7-494b-4050-8a5a-8a604bf056c4-kube-api-access-j49rs\") pod \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\" (UID: \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\") "
Nov 02 07:40:08.804865 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.804479    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-openshift-ca\") pod \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\" (UID: \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\") "
Nov 02 07:40:08.804865 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.804517    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-xks-ca\") pod \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\" (UID: \"1cb4ced7-494b-4050-8a5a-8a604bf056c4\") "
Nov 02 07:40:08.804865 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.804806    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-openshift-ca" pod "1cb4ced7-494b-4050-8a5a-8a604bf056c4" (UID: "1cb4ced7-494b-4050-8a5a-8a604bf056c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:08.805001 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.804974    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:08.806686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.806662    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb4ced7-494b-4050-8a5a-8a604bf056c4-kube-api-access-j49rs" pod "1cb4ced7-494b-4050-8a5a-8a604bf056c4" (UID: "1cb4ced7-494b-4050-8a5a-8a604bf056c4"). InnerVolumeSpecName "kube-api-access-j49rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:08.806686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.806667    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-xks-ca" pod "1cb4ced7-494b-4050-8a5a-8a604bf056c4" (UID: "1cb4ced7-494b-4050-8a5a-8a604bf056c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:40:08.906690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.906646    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb4ced7-494b-4050-8a5a-8a604bf056c4-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:08.906690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:08.906678    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j49rs\" (UniqueName: \"kubernetes.io/projected/1cb4ced7-494b-4050-8a5a-8a604bf056c4-kube-api-access-j49rs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:09.515577 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:09.515546    4898 generic.go:356] "Generic (PLEG): container finished" podID="da36ae40-a2c8-432a-bc54-6c1097e97b1a" containerID="7a75bb67920e0be74b802d7c0b97efb8e7123bff9b42d96e41329970d8dfd674" exitCode=0
Nov 02 07:40:09.515746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:09.515625    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7" event={"ID":"da36ae40-a2c8-432a-bc54-6c1097e97b1a","Type":"ContainerDied","Data":"7a75bb67920e0be74b802d7c0b97efb8e7123bff9b42d96e41329970d8dfd674"}
Nov 02 07:40:09.517268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:09.517250    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"
Nov 02 07:40:09.517403 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:09.517276    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj" event={"ID":"1cb4ced7-494b-4050-8a5a-8a604bf056c4","Type":"ContainerDied","Data":"f7c4ebc07586ac90240be544ee5abc33b5beddf34e2b414715b50b2f20fd4f96"}
Nov 02 07:40:09.517403 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:09.517303    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7c4ebc07586ac90240be544ee5abc33b5beddf34e2b414715b50b2f20fd4f96"
Nov 02 07:40:10.685016 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:10.684992    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:10.837837 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:10.837747    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-xks-ca\") pod \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\" (UID: \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\") "
Nov 02 07:40:10.837995 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:10.837960    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da36ae40-a2c8-432a-bc54-6c1097e97b1a-kube-api-access-xzsjf\" (UniqueName: \"kubernetes.io/projected/da36ae40-a2c8-432a-bc54-6c1097e97b1a-kube-api-access-xzsjf\") pod \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\" (UID: \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\") "
Nov 02 07:40:10.838086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:10.838045    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-openshift-ca\") pod \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\" (UID: \"da36ae40-a2c8-432a-bc54-6c1097e97b1a\") "
Nov 02 07:40:10.838537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:10.838501    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-openshift-ca" pod "da36ae40-a2c8-432a-bc54-6c1097e97b1a" (UID: "da36ae40-a2c8-432a-bc54-6c1097e97b1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:10.839979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:10.839947    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-xks-ca" pod "da36ae40-a2c8-432a-bc54-6c1097e97b1a" (UID: "da36ae40-a2c8-432a-bc54-6c1097e97b1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:40:10.840228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:10.840207    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da36ae40-a2c8-432a-bc54-6c1097e97b1a-kube-api-access-xzsjf" pod "da36ae40-a2c8-432a-bc54-6c1097e97b1a" (UID: "da36ae40-a2c8-432a-bc54-6c1097e97b1a"). InnerVolumeSpecName "kube-api-access-xzsjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:10.939459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:10.939425    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:10.939459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:10.939455    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzsjf\" (UniqueName: \"kubernetes.io/projected/da36ae40-a2c8-432a-bc54-6c1097e97b1a-kube-api-access-xzsjf\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:10.939459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:10.939467    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da36ae40-a2c8-432a-bc54-6c1097e97b1a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:11.470189 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:11.470154    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"]
Nov 02 07:40:11.479920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:11.479716    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:40:11.535093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:11.535055    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7" event={"ID":"da36ae40-a2c8-432a-bc54-6c1097e97b1a","Type":"ContainerDied","Data":"61d14c83eafed65b4a1b30583ff3e79fcc243134ae355b4d5430891a6c50db15"}
Nov 02 07:40:11.535093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:11.535094    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"
Nov 02 07:40:11.535093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:11.535115    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61d14c83eafed65b4a1b30583ff3e79fcc243134ae355b4d5430891a6c50db15"
Nov 02 07:40:12.071964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.071929    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"]
Nov 02 07:40:12.072725 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.072703    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1cb4ced7-494b-4050-8a5a-8a604bf056c4" containerName="revoke-keys"
Nov 02 07:40:12.072725 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.072725    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb4ced7-494b-4050-8a5a-8a604bf056c4" containerName="revoke-keys"
Nov 02 07:40:12.072921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.072756    4898 cpu_manager.go:414] "containerMap: removing container" podUID="da36ae40-a2c8-432a-bc54-6c1097e97b1a" containerName="revoke-keys"
Nov 02 07:40:12.072921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.072764    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da36ae40-a2c8-432a-bc54-6c1097e97b1a" containerName="revoke-keys"
Nov 02 07:40:12.073016 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.072962    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb4ced7-494b-4050-8a5a-8a604bf056c4" containerName="revoke-keys"
Nov 02 07:40:12.073016 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.072979    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="da36ae40-a2c8-432a-bc54-6c1097e97b1a" containerName="revoke-keys"
Nov 02 07:40:12.076934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.076914    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.088201 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.088175    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"]
Nov 02 07:40:12.092319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.092300    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.103771 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.103745    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"]
Nov 02 07:40:12.122539 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.122115    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"]
Nov 02 07:40:12.159169 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159137    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.159321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159180    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.159321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159210    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.159321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159235    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.159321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159274    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.159321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159313    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.159574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159358    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.159574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159408    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.159574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159434    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.159574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159492    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brl85\" (UniqueName: \"kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-kube-api-access-brl85\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.159574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159557    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.160244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159589    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.160244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159616    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.160244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159642    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.160244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159718    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xzw7\" (UniqueName: \"kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-kube-api-access-5xzw7\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.160244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159791    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.160244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159840    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.160244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.159863    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbe2608e-0a74-4371-9bc7-8425fb955660-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.261527 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.261493    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.261689 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.261555    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.261689 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.261603    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.261689 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.261632    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.261882 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.261684    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brl85\" (UniqueName: \"kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-kube-api-access-brl85\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.261965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.261898    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.261965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.261932    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.261965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.261947    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.262129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.261995    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.262129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.261995    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.262129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262022    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.262129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262025    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.262129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262102    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xzw7\" (UniqueName: \"kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-kube-api-access-5xzw7\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.262392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262185    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.262392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262229    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.262392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262253    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbe2608e-0a74-4371-9bc7-8425fb955660-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.262392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262334    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.262392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262364    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.262657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262392    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.262657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262424    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.262657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262453    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.262657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262468    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.262657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262621    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.262923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262766    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-certs\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.262923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.262778    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.263131 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.263104    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-data\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.263192 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.263160    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-credential-socket\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.263440 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.263417    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbe2608e-0a74-4371-9bc7-8425fb955660-istiod-ca-cert\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.264697 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.264665    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.264954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.264936    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.265108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.265090    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-podinfo\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.265206 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.265190    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-envoy\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.284765 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.284738    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.285352 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.285324    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brl85\" (UniqueName: \"kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-kube-api-access-brl85\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.286038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.285992    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-token\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.286212 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.286188    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xzw7\" (UniqueName: \"kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-kube-api-access-5xzw7\") pod \"e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") " pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.394091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.394024    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:12.412830 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.412805    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:12.713008 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:12.712975    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5377b5_78d2_4523_b7cc_6eb386e15ad5.slice/crio-ee8e9cc9b09a9be4a16d74bc91efec23170cf8373a3fe99560046316de1522cb WatchSource:0}: Error finding container ee8e9cc9b09a9be4a16d74bc91efec23170cf8373a3fe99560046316de1522cb: Status 404 returned error can't find the container with id ee8e9cc9b09a9be4a16d74bc91efec23170cf8373a3fe99560046316de1522cb
Nov 02 07:40:12.716514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.716469    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:12.716806 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.716769    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:12.716921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.716876    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:12.718991 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.718964    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"]
Nov 02 07:40:12.738714 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:12.738685    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe2608e_0a74_4371_9bc7_8425fb955660.slice/crio-afaa76bbfc1ea7e5d21ec7b22c8add241cb5c04eea186eb6c6c30ab96be87d29 WatchSource:0}: Error finding container afaa76bbfc1ea7e5d21ec7b22c8add241cb5c04eea186eb6c6c30ab96be87d29: Status 404 returned error can't find the container with id afaa76bbfc1ea7e5d21ec7b22c8add241cb5c04eea186eb6c6c30ab96be87d29
Nov 02 07:40:12.740926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.740880    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"]
Nov 02 07:40:12.741487 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.741460    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:12.741548 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.741517    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:12.741548 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:12.741543    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:13.618190 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:13.617402    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" event={"ID":"dbe2608e-0a74-4371-9bc7-8425fb955660","Type":"ContainerStarted","Data":"3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738"}
Nov 02 07:40:13.618190 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:13.617472    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" event={"ID":"dbe2608e-0a74-4371-9bc7-8425fb955660","Type":"ContainerStarted","Data":"afaa76bbfc1ea7e5d21ec7b22c8add241cb5c04eea186eb6c6c30ab96be87d29"}
Nov 02 07:40:13.622670 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:13.622624    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" event={"ID":"eb5377b5-78d2-4523-b7cc-6eb386e15ad5","Type":"ContainerStarted","Data":"bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5"}
Nov 02 07:40:13.622796 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:13.622703    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" event={"ID":"eb5377b5-78d2-4523-b7cc-6eb386e15ad5","Type":"ContainerStarted","Data":"ee8e9cc9b09a9be4a16d74bc91efec23170cf8373a3fe99560046316de1522cb"}
Nov 02 07:40:13.750052 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:13.749986    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" podStartSLOduration=1.74997185 podStartE2EDuration="1.74997185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:13.747586767 +0000 UTC m=+964.437913751" watchObservedRunningTime="2026-11-02 07:40:13.74997185 +0000 UTC m=+964.440298836"
Nov 02 07:40:13.751936 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:13.751882    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" podStartSLOduration=1.75187289 podStartE2EDuration="1.75187289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:13.702282031 +0000 UTC m=+964.392609021" watchObservedRunningTime="2026-11-02 07:40:13.75187289 +0000 UTC m=+964.442199896"
Nov 02 07:40:14.398356 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:14.398317    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:14.416567 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:14.416529    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 07:40:14.416827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:14.416795    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 07:40:14.424228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:14.417385    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:14.426039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:14.426015    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 07:40:14.426216 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:14.426192    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" podUID="dbe2608e-0a74-4371-9bc7-8425fb955660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 07:40:15.033650 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:40:15.033613    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:40:15.395083 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.394989    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 07:40:15.395434 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.395405    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 07:40:15.423928 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.423032    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 07:40:15.423928 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.423108    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" podUID="dbe2608e-0a74-4371-9bc7-8425fb955660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 07:40:15.846581 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.846551    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"]
Nov 02 07:40:15.851580 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.851558    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:15.888741 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.888712    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:40:15.903252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.903125    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:40:15.903252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.903224    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:40:15.903590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.903572    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:40:15.943283 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.943234    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"]
Nov 02 07:40:15.947732 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.947698    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd5bd9da-6znh9\" (UID: \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:15.947932 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.947905    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g8nd\" (UniqueName: \"kubernetes.io/projected/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-kube-api-access-6g8nd\") pod \"maas-api-revoke-sub-cd5bd9da-6znh9\" (UID: \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:15.948005 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:15.947960    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd5bd9da-6znh9\" (UID: \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:16.049385 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.049341    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd5bd9da-6znh9\" (UID: \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:16.049773 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.049436    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g8nd\" (UniqueName: \"kubernetes.io/projected/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-kube-api-access-6g8nd\") pod \"maas-api-revoke-sub-cd5bd9da-6znh9\" (UID: \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:16.049773 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.049461    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd5bd9da-6znh9\" (UID: \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:16.050183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.050163    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd5bd9da-6znh9\" (UID: \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:16.051734 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.051710    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd5bd9da-6znh9\" (UID: \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:16.118294 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.118215    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g8nd\" (UniqueName: \"kubernetes.io/projected/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-kube-api-access-6g8nd\") pod \"maas-api-revoke-sub-cd5bd9da-6znh9\" (UID: \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:16.208913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.208133    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:16.426398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.425470    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 07:40:16.426398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.425567    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" podUID="dbe2608e-0a74-4371-9bc7-8425fb955660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 07:40:16.426398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.425977    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 07:40:16.426398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.426034    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 07:40:16.508956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.499958    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f" podUID="7e04da48-887d-47f1-960c-ae268b1c7586" containerName="payload-processing" containerID="cri-o://c444742c42adfc88b5e68224b1ff8508bf1c188374738f78ccebfb6428607af0" gracePeriod=25
Nov 02 07:40:16.622856 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.622810    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"]
Nov 02 07:40:16.632045 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:16.632005    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5325aca2_4bb7_4b29_9c83_fe4ce9bcd410.slice/crio-0a3fb4dea2ad9ba6400ec81eef59685f41d14d07c1d14e16f1686306e6fe0b19 WatchSource:0}: Error finding container 0a3fb4dea2ad9ba6400ec81eef59685f41d14d07c1d14e16f1686306e6fe0b19: Status 404 returned error can't find the container with id 0a3fb4dea2ad9ba6400ec81eef59685f41d14d07c1d14e16f1686306e6fe0b19
Nov 02 07:40:16.666102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.666012    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9" event={"ID":"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410","Type":"ContainerStarted","Data":"0a3fb4dea2ad9ba6400ec81eef59685f41d14d07c1d14e16f1686306e6fe0b19"}
Nov 02 07:40:16.674986 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.674830    4898 generic.go:356] "Generic (PLEG): container finished" podID="7e04da48-887d-47f1-960c-ae268b1c7586" containerID="c444742c42adfc88b5e68224b1ff8508bf1c188374738f78ccebfb6428607af0" exitCode=0
Nov 02 07:40:16.674986 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:16.674920    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f" event={"ID":"7e04da48-887d-47f1-960c-ae268b1c7586","Type":"ContainerDied","Data":"c444742c42adfc88b5e68224b1ff8508bf1c188374738f78ccebfb6428607af0"}
Nov 02 07:40:17.050166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.050050    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:40:17.149914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.146282    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-routing-overlay-volume\") pod \"7e04da48-887d-47f1-960c-ae268b1c7586\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") "
Nov 02 07:40:17.149914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.146512    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-kube-api-access-65825\" (UniqueName: \"kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-kube-api-access-65825\") pod \"7e04da48-887d-47f1-960c-ae268b1c7586\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") "
Nov 02 07:40:17.149914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.146582    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-plugins-config-volume\") pod \"7e04da48-887d-47f1-960c-ae268b1c7586\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") "
Nov 02 07:40:17.149914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.146675    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-provider-credentials\") pod \"7e04da48-887d-47f1-960c-ae268b1c7586\" (UID: \"7e04da48-887d-47f1-960c-ae268b1c7586\") "
Nov 02 07:40:17.151038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.150981    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-plugins-config-volume" pod "7e04da48-887d-47f1-960c-ae268b1c7586" (UID: "7e04da48-887d-47f1-960c-ae268b1c7586"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:17.152471 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.152451    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-routing-overlay-volume" pod "7e04da48-887d-47f1-960c-ae268b1c7586" (UID: "7e04da48-887d-47f1-960c-ae268b1c7586"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:17.152986 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.152967    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-provider-credentials" pod "7e04da48-887d-47f1-960c-ae268b1c7586" (UID: "7e04da48-887d-47f1-960c-ae268b1c7586"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:17.153796 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.153756    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-kube-api-access-65825" pod "7e04da48-887d-47f1-960c-ae268b1c7586" (UID: "7e04da48-887d-47f1-960c-ae268b1c7586"). InnerVolumeSpecName "kube-api-access-65825". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:17.248512 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.248475    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65825\" (UniqueName: \"kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-kube-api-access-65825\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:17.248512 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.248505    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:17.248512 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.248515    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e04da48-887d-47f1-960c-ae268b1c7586-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:17.248735 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.248525    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e04da48-887d-47f1-960c-ae268b1c7586-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:17.395314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.395220    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 07:40:17.395998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.395564    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 07:40:17.419922 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.416997    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 07:40:17.419922 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.417052    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" podUID="dbe2608e-0a74-4371-9bc7-8425fb955660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 07:40:17.542377 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.542344    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"]
Nov 02 07:40:17.542928 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.542914    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7e04da48-887d-47f1-960c-ae268b1c7586" containerName="payload-processing"
Nov 02 07:40:17.542979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.542930    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e04da48-887d-47f1-960c-ae268b1c7586" containerName="payload-processing"
Nov 02 07:40:17.543070 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.543061    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e04da48-887d-47f1-960c-ae268b1c7586" containerName="payload-processing"
Nov 02 07:40:17.546668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.546652    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:17.634721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.634680    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"]
Nov 02 07:40:17.652832 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.652775    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prq6n\" (UniqueName: \"kubernetes.io/projected/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-kube-api-access-prq6n\") pod \"maas-api-revoke-sub-0c93d984-v7qvk\" (UID: \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:17.652993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.652906    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c93d984-v7qvk\" (UID: \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:17.652993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.652959    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c93d984-v7qvk\" (UID: \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:17.722913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.722543    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"
Nov 02 07:40:17.723119 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.722968    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f" event={"ID":"7e04da48-887d-47f1-960c-ae268b1c7586","Type":"ContainerDied","Data":"cdc89798b50a13aee94e105b89f19c266a382968d7b830cd2b25d97a8d7fb561"}
Nov 02 07:40:17.723119 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.723030    4898 scope.go:122] "RemoveContainer" containerID="c444742c42adfc88b5e68224b1ff8508bf1c188374738f78ccebfb6428607af0"
Nov 02 07:40:17.725795 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.725770    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9" event={"ID":"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410","Type":"ContainerStarted","Data":"1a4cc0214e91b58a4bcee979323abc4be4916bcff9793d5741e3c2dc46c970a7"}
Nov 02 07:40:17.753593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.753567    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c93d984-v7qvk\" (UID: \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:17.753742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.753628    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c93d984-v7qvk\" (UID: \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:17.753742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.753733    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prq6n\" (UniqueName: \"kubernetes.io/projected/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-kube-api-access-prq6n\") pod \"maas-api-revoke-sub-0c93d984-v7qvk\" (UID: \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:17.754521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.754497    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c93d984-v7qvk\" (UID: \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:17.807654 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.807237    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"]
Nov 02 07:40:17.828367 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.828323    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c93d984-v7qvk\" (UID: \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:17.829026 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.829004    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prq6n\" (UniqueName: \"kubernetes.io/projected/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-kube-api-access-prq6n\") pod \"maas-api-revoke-sub-0c93d984-v7qvk\" (UID: \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:17.832120 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.832090    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-66bbdd4b74-mfg7f"]
Nov 02 07:40:17.875922 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.868784    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9" podStartSLOduration=2.8687660839999998 podStartE2EDuration="2.868766084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:17.866335574 +0000 UTC m=+968.556662559" watchObservedRunningTime="2026-11-02 07:40:17.868766084 +0000 UTC m=+968.559093049"
Nov 02 07:40:17.915986 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:17.913068    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:18.320571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:18.320025    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"]
Nov 02 07:40:18.323065 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:18.323038    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b6cf31_1501_4b1e_b5d0_bc4e391c8189.slice/crio-759745edd5d964d932d443b5f8202099831d0d30cf00a9f9531aabe65f11711e WatchSource:0}: Error finding container 759745edd5d964d932d443b5f8202099831d0d30cf00a9f9531aabe65f11711e: Status 404 returned error can't find the container with id 759745edd5d964d932d443b5f8202099831d0d30cf00a9f9531aabe65f11711e
Nov 02 07:40:18.395611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:18.395579    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 07:40:18.395836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:18.395811    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 07:40:18.421505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:18.421474    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 07:40:18.421731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:18.421704    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" podUID="dbe2608e-0a74-4371-9bc7-8425fb955660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 07:40:18.736725 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:18.736687    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk" event={"ID":"02b6cf31-1501-4b1e-b5d0-bc4e391c8189","Type":"ContainerStarted","Data":"759745edd5d964d932d443b5f8202099831d0d30cf00a9f9531aabe65f11711e"}
Nov 02 07:40:18.738002 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:18.737977    4898 generic.go:356] "Generic (PLEG): container finished" podID="5325aca2-4bb7-4b29-9c83-fe4ce9bcd410" containerID="1a4cc0214e91b58a4bcee979323abc4be4916bcff9793d5741e3c2dc46c970a7" exitCode=0
Nov 02 07:40:18.738142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:18.738026    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9" event={"ID":"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410","Type":"ContainerDied","Data":"1a4cc0214e91b58a4bcee979323abc4be4916bcff9793d5741e3c2dc46c970a7"}
Nov 02 07:40:19.395155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:19.395116    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 07:40:19.395854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:19.395819    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 07:40:19.414213 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:19.414169    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 02 07:40:19.414357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:19.414234    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" podUID="dbe2608e-0a74-4371-9bc7-8425fb955660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 02 07:40:19.492741 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:19.492699    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e04da48-887d-47f1-960c-ae268b1c7586" path="/var/lib/kubelet/pods/7e04da48-887d-47f1-960c-ae268b1c7586/volumes"
Nov 02 07:40:19.753111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:19.753071    4898 generic.go:356] "Generic (PLEG): container finished" podID="02b6cf31-1501-4b1e-b5d0-bc4e391c8189" containerID="14d3f8d805631699924c37220646604ba12535b01542fa9bd479e15b4abb8646" exitCode=0
Nov 02 07:40:19.753604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:19.753577    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk" event={"ID":"02b6cf31-1501-4b1e-b5d0-bc4e391c8189","Type":"ContainerDied","Data":"14d3f8d805631699924c37220646604ba12535b01542fa9bd479e15b4abb8646"}
Nov 02 07:40:20.296914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.295086    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:20.607917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.563558    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-kube-api-access-6g8nd\" (UniqueName: \"kubernetes.io/projected/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-kube-api-access-6g8nd\") pod \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\" (UID: \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\") "
Nov 02 07:40:20.607917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.563619    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-openshift-ca\") pod \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\" (UID: \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\") "
Nov 02 07:40:20.607917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.563758    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-xks-ca\") pod \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\" (UID: \"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410\") "
Nov 02 07:40:20.680914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.667403    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-openshift-ca" pod "5325aca2-4bb7-4b29-9c83-fe4ce9bcd410" (UID: "5325aca2-4bb7-4b29-9c83-fe4ce9bcd410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:20.680914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.668586    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:20.914976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.906421    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-kube-api-access-6g8nd" pod "5325aca2-4bb7-4b29-9c83-fe4ce9bcd410" (UID: "5325aca2-4bb7-4b29-9c83-fe4ce9bcd410"). InnerVolumeSpecName "kube-api-access-6g8nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:20.949139 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.948847    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"
Nov 02 07:40:20.949139 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.949064    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9" event={"ID":"5325aca2-4bb7-4b29-9c83-fe4ce9bcd410","Type":"ContainerDied","Data":"0a3fb4dea2ad9ba6400ec81eef59685f41d14d07c1d14e16f1686306e6fe0b19"}
Nov 02 07:40:20.949139 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.949111    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a3fb4dea2ad9ba6400ec81eef59685f41d14d07c1d14e16f1686306e6fe0b19"
Nov 02 07:40:20.966165 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.966058    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-xks-ca" pod "5325aca2-4bb7-4b29-9c83-fe4ce9bcd410" (UID: "5325aca2-4bb7-4b29-9c83-fe4ce9bcd410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:40:20.991553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.991473    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g8nd\" (UniqueName: \"kubernetes.io/projected/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-kube-api-access-6g8nd\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:20.991553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:20.991513    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:21.460916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:21.453006    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:40:21.460916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:21.456952    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:21.470134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:21.470080    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:40:21.493974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:21.470393    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" podUID="dbe2608e-0a74-4371-9bc7-8425fb955660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:21.714954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:21.713228    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:21.714954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:21.714760    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:21.721232 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:21.721198    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 07:40:21.985688 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:21.985596    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:22.006950 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:21.998533    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:22.021184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.021090    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:40:22.021614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.021536    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:22.142901 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.142683    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"
Nov 02 07:40:22.142901 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.142815    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk" event={"ID":"02b6cf31-1501-4b1e-b5d0-bc4e391c8189","Type":"ContainerDied","Data":"759745edd5d964d932d443b5f8202099831d0d30cf00a9f9531aabe65f11711e"}
Nov 02 07:40:22.142901 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.142850    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="759745edd5d964d932d443b5f8202099831d0d30cf00a9f9531aabe65f11711e"
Nov 02 07:40:22.163055 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.161966    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 02 07:40:22.198519 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.198485    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-kube-api-access-prq6n\" (UniqueName: \"kubernetes.io/projected/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-kube-api-access-prq6n\") pod \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\" (UID: \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\") "
Nov 02 07:40:22.205093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.205053    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-openshift-ca\") pod \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\" (UID: \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\") "
Nov 02 07:40:22.205385 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.205358    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-xks-ca\") pod \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\" (UID: \"02b6cf31-1501-4b1e-b5d0-bc4e391c8189\") "
Nov 02 07:40:22.214912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.198762    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:22.244962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.219394    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-openshift-ca" pod "02b6cf31-1501-4b1e-b5d0-bc4e391c8189" (UID: "02b6cf31-1501-4b1e-b5d0-bc4e391c8189"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:22.310041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.310005    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:22.326027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.325981    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:22.342916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.333747    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:22.396915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.382923    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:22.396915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.383986    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:22.398297 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.398262    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:22.531624 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.531520    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:22.537914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.531976    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-kube-api-access-prq6n" pod "02b6cf31-1501-4b1e-b5d0-bc4e391c8189" (UID: "02b6cf31-1501-4b1e-b5d0-bc4e391c8189"). InnerVolumeSpecName "kube-api-access-prq6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:22.646391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.568667    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prq6n\" (UniqueName: \"kubernetes.io/projected/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-kube-api-access-prq6n\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:22.696916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.680256    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-xks-ca" pod "02b6cf31-1501-4b1e-b5d0-bc4e391c8189" (UID: "02b6cf31-1501-4b1e-b5d0-bc4e391c8189"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:40:22.801913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:22.798165    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b6cf31-1501-4b1e-b5d0-bc4e391c8189-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:23.164039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.163966    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:23.167919 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.164429    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:40:23.277025 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.257048    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:40:23.277025 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.257720    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:40:23.327129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.327019    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:23.350945 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.328221    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:40:23.489011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.383571    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:23.489011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.384033    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:40:23.489011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.385228    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:23.489011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.385580    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:40:23.489011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.478355    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:23.601915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.539796    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 07:40:23.705962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:23.543631    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:24.141956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:24.118636    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:40:24.141956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:24.118707    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:40:24.165793 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:24.165234    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:24.300919 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:24.257730    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:24.376811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:24.374859    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:24.426377 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:24.426334    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 02 07:40:24.430912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:24.426923    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:24.522914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:24.517031    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:24.522914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:24.518163    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:40:25.040441 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:25.034971    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:25.062395 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:25.061158    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:25.168263 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:25.166385    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:25.322186 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:25.314260    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:40:25.366707 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:25.330870    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded"
Nov 02 07:40:25.394462 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:25.393422    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:25.441032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:25.438350    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:25.471914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:25.471365    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:25.533915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:25.531969    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:40:26.040126 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.038966    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:26.089134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.074322    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:26.220028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.219979    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:26.358082 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.357977    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:26.428913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.400550    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:26.431073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.425426    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:40:26.442672 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.442636    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:26.614648 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.614554    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:26.650929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.650584    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:40:26.692912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.658144    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:40:26.806173 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.805867    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:40:26.812780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.812742    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:40:26.841045 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.840994    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:40:26.923640 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.923149    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:40:26.934923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:26.934131    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:40:28.193719 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.193673    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"]
Nov 02 07:40:28.194268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.194254    4898 cpu_manager.go:414] "containerMap: removing container" podUID="5325aca2-4bb7-4b29-9c83-fe4ce9bcd410" containerName="revoke-keys"
Nov 02 07:40:28.194336 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.194270    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5325aca2-4bb7-4b29-9c83-fe4ce9bcd410" containerName="revoke-keys"
Nov 02 07:40:28.194336 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.194308    4898 cpu_manager.go:414] "containerMap: removing container" podUID="02b6cf31-1501-4b1e-b5d0-bc4e391c8189" containerName="revoke-keys"
Nov 02 07:40:28.194336 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.194313    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b6cf31-1501-4b1e-b5d0-bc4e391c8189" containerName="revoke-keys"
Nov 02 07:40:28.194432 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.194394    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b6cf31-1501-4b1e-b5d0-bc4e391c8189" containerName="revoke-keys"
Nov 02 07:40:28.194432 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.194405    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5325aca2-4bb7-4b29-9c83-fe4ce9bcd410" containerName="revoke-keys"
Nov 02 07:40:28.198109 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.198087    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:28.215671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.215645    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"]
Nov 02 07:40:28.223356 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.223321    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:40:28.225011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.224987    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:40:28.225131 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.225019    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:40:28.225131 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.225054    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:40:28.298191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.298135    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c78ed9d-4wxwl\" (UID: \"70ec3fd5-230c-4ce6-b093-814e6ef74954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:28.298358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.298338    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5s5s\" (UniqueName: \"kubernetes.io/projected/70ec3fd5-230c-4ce6-b093-814e6ef74954-kube-api-access-d5s5s\") pod \"maas-api-revoke-sub-2c78ed9d-4wxwl\" (UID: \"70ec3fd5-230c-4ce6-b093-814e6ef74954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:28.298404 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.298387    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c78ed9d-4wxwl\" (UID: \"70ec3fd5-230c-4ce6-b093-814e6ef74954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:28.400156 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.400114    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c78ed9d-4wxwl\" (UID: \"70ec3fd5-230c-4ce6-b093-814e6ef74954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:28.400377 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.400276    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5s5s\" (UniqueName: \"kubernetes.io/projected/70ec3fd5-230c-4ce6-b093-814e6ef74954-kube-api-access-d5s5s\") pod \"maas-api-revoke-sub-2c78ed9d-4wxwl\" (UID: \"70ec3fd5-230c-4ce6-b093-814e6ef74954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:28.400377 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.400334    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c78ed9d-4wxwl\" (UID: \"70ec3fd5-230c-4ce6-b093-814e6ef74954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:28.400993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.400966    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c78ed9d-4wxwl\" (UID: \"70ec3fd5-230c-4ce6-b093-814e6ef74954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:28.403411 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.403381    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c78ed9d-4wxwl\" (UID: \"70ec3fd5-230c-4ce6-b093-814e6ef74954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:28.425099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.425057    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5s5s\" (UniqueName: \"kubernetes.io/projected/70ec3fd5-230c-4ce6-b093-814e6ef74954-kube-api-access-d5s5s\") pod \"maas-api-revoke-sub-2c78ed9d-4wxwl\" (UID: \"70ec3fd5-230c-4ce6-b093-814e6ef74954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:28.513248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.513159    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:28.818940 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:28.818876    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ec3fd5_230c_4ce6_b093_814e6ef74954.slice/crio-b7e22d2bd4034cb2f0c445a6e8a50a29b0169d227a86b685becaca14379c479b WatchSource:0}: Error finding container b7e22d2bd4034cb2f0c445a6e8a50a29b0169d227a86b685becaca14379c479b: Status 404 returned error can't find the container with id b7e22d2bd4034cb2f0c445a6e8a50a29b0169d227a86b685becaca14379c479b
Nov 02 07:40:28.821210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.821186    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"]
Nov 02 07:40:28.967094 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:28.967061    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl" event={"ID":"70ec3fd5-230c-4ce6-b093-814e6ef74954","Type":"ContainerStarted","Data":"b7e22d2bd4034cb2f0c445a6e8a50a29b0169d227a86b685becaca14379c479b"}
Nov 02 07:40:29.103784 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.103748    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"]
Nov 02 07:40:29.110256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.110229    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.125915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.125863    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"]
Nov 02 07:40:29.131671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.131607    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9db802\"/\"default-dockercfg-pnzx7\""
Nov 02 07:40:29.131671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.131632    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9db802\"/\"distinct-w4-9db802-kserve-self-signed-certs\""
Nov 02 07:40:29.137116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.137042    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9db802\"/\"openshift-service-ca.crt\""
Nov 02 07:40:29.137224 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.137191    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9db802\"/\"kube-root-ca.crt\""
Nov 02 07:40:29.210798 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.210766    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-tmp-dir\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.211264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.210806    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-model-cache\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.211264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.210961    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1947df5d-7c32-4906-9e5b-266da776ed8b-tls-certs\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.211264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.211015    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-dshm\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.211264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.211143    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsmgm\" (UniqueName: \"kubernetes.io/projected/1947df5d-7c32-4906-9e5b-266da776ed8b-kube-api-access-tsmgm\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.211264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.211212    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-home\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.313429 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.313393    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1947df5d-7c32-4906-9e5b-266da776ed8b-tls-certs\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.313634 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.313440    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-dshm\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.313634 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.313496    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsmgm\" (UniqueName: \"kubernetes.io/projected/1947df5d-7c32-4906-9e5b-266da776ed8b-kube-api-access-tsmgm\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.313634 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.313559    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-home\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.313634 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.313628    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-tmp-dir\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.313834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.313663    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-model-cache\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.314470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.314026    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-tmp-dir\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.314470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.314216    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-model-cache\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.314657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.314582    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-home\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.316049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.316026    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-dshm\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.316227 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.316208    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1947df5d-7c32-4906-9e5b-266da776ed8b-tls-certs\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.337492 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.337440    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsmgm\" (UniqueName: \"kubernetes.io/projected/1947df5d-7c32-4906-9e5b-266da776ed8b-kube-api-access-tsmgm\") pod \"distinct-w4-9db802-kserve-5957f8db48-hfd6j\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") " pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.438801 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.438767    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:29.761505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.761464    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"]
Nov 02 07:40:29.766581 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:29.766545    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1947df5d_7c32_4906_9e5b_266da776ed8b.slice/crio-39a54ec4990daf30fcc8149444d67876e80f6f9a4204ce2ab29d406a3815bee4 WatchSource:0}: Error finding container 39a54ec4990daf30fcc8149444d67876e80f6f9a4204ce2ab29d406a3815bee4: Status 404 returned error can't find the container with id 39a54ec4990daf30fcc8149444d67876e80f6f9a4204ce2ab29d406a3815bee4
Nov 02 07:40:29.977312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.977275    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" event={"ID":"1947df5d-7c32-4906-9e5b-266da776ed8b","Type":"ContainerStarted","Data":"93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98"}
Nov 02 07:40:29.977488 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.977324    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" event={"ID":"1947df5d-7c32-4906-9e5b-266da776ed8b","Type":"ContainerStarted","Data":"39a54ec4990daf30fcc8149444d67876e80f6f9a4204ce2ab29d406a3815bee4"}
Nov 02 07:40:29.978945 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.978915    4898 generic.go:356] "Generic (PLEG): container finished" podID="70ec3fd5-230c-4ce6-b093-814e6ef74954" containerID="baccb7d67f44ae37f0dea7d29fae07769b7442ddca0468127d7a725fe5c50894" exitCode=0
Nov 02 07:40:29.979064 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:29.978946    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl" event={"ID":"70ec3fd5-230c-4ce6-b093-814e6ef74954","Type":"ContainerDied","Data":"baccb7d67f44ae37f0dea7d29fae07769b7442ddca0468127d7a725fe5c50894"}
Nov 02 07:40:30.020779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.020691    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podStartSLOduration=1.020676501 podStartE2EDuration="1.020676501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:30.017457613 +0000 UTC m=+980.707784590" watchObservedRunningTime="2026-11-02 07:40:30.020676501 +0000 UTC m=+980.711003469"
Nov 02 07:40:30.111544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.111516    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"]
Nov 02 07:40:30.115815 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.115788    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:30.137991 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.137943    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"]
Nov 02 07:40:30.143530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.143499    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1022cc-7xr76\" (UID: \"90794722-dc5a-4d3c-9756-5584bb04d93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:30.143861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.143838    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrmt\" (UniqueName: \"kubernetes.io/projected/90794722-dc5a-4d3c-9756-5584bb04d93c-kube-api-access-htrmt\") pod \"maas-api-revoke-sub-2e1022cc-7xr76\" (UID: \"90794722-dc5a-4d3c-9756-5584bb04d93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:30.144204 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.144178    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1022cc-7xr76\" (UID: \"90794722-dc5a-4d3c-9756-5584bb04d93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:30.245583 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.245547    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrmt\" (UniqueName: \"kubernetes.io/projected/90794722-dc5a-4d3c-9756-5584bb04d93c-kube-api-access-htrmt\") pod \"maas-api-revoke-sub-2e1022cc-7xr76\" (UID: \"90794722-dc5a-4d3c-9756-5584bb04d93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:30.246092 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.245709    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1022cc-7xr76\" (UID: \"90794722-dc5a-4d3c-9756-5584bb04d93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:30.246092 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.245798    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1022cc-7xr76\" (UID: \"90794722-dc5a-4d3c-9756-5584bb04d93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:30.246507 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.246481    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1022cc-7xr76\" (UID: \"90794722-dc5a-4d3c-9756-5584bb04d93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:30.248187 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.248158    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1022cc-7xr76\" (UID: \"90794722-dc5a-4d3c-9756-5584bb04d93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:30.268498 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.268474    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrmt\" (UniqueName: \"kubernetes.io/projected/90794722-dc5a-4d3c-9756-5584bb04d93c-kube-api-access-htrmt\") pod \"maas-api-revoke-sub-2e1022cc-7xr76\" (UID: \"90794722-dc5a-4d3c-9756-5584bb04d93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:30.430284 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.430253    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:30.439006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.438978    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:30.751948 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:30.751923    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90794722_dc5a_4d3c_9756_5584bb04d93c.slice/crio-f1232e08927bec9f644071e0c8837e894b17f0955d1349d8fed56e0ed211be09 WatchSource:0}: Error finding container f1232e08927bec9f644071e0c8837e894b17f0955d1349d8fed56e0ed211be09: Status 404 returned error can't find the container with id f1232e08927bec9f644071e0c8837e894b17f0955d1349d8fed56e0ed211be09
Nov 02 07:40:30.754959 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.754932    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"]
Nov 02 07:40:30.988990 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:30.988954    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76" event={"ID":"90794722-dc5a-4d3c-9756-5584bb04d93c","Type":"ContainerStarted","Data":"f1232e08927bec9f644071e0c8837e894b17f0955d1349d8fed56e0ed211be09"}
Nov 02 07:40:31.123158 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:31.123133    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:31.163304 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:31.163273    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-xks-ca\") pod \"70ec3fd5-230c-4ce6-b093-814e6ef74954\" (UID: \"70ec3fd5-230c-4ce6-b093-814e6ef74954\") "
Nov 02 07:40:31.163483 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:31.163465    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ec3fd5-230c-4ce6-b093-814e6ef74954-kube-api-access-d5s5s\" (UniqueName: \"kubernetes.io/projected/70ec3fd5-230c-4ce6-b093-814e6ef74954-kube-api-access-d5s5s\") pod \"70ec3fd5-230c-4ce6-b093-814e6ef74954\" (UID: \"70ec3fd5-230c-4ce6-b093-814e6ef74954\") "
Nov 02 07:40:31.163604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:31.163589    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-openshift-ca\") pod \"70ec3fd5-230c-4ce6-b093-814e6ef74954\" (UID: \"70ec3fd5-230c-4ce6-b093-814e6ef74954\") "
Nov 02 07:40:31.164715 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:31.164676    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-openshift-ca" pod "70ec3fd5-230c-4ce6-b093-814e6ef74954" (UID: "70ec3fd5-230c-4ce6-b093-814e6ef74954"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:31.167880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:31.167848    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-xks-ca" pod "70ec3fd5-230c-4ce6-b093-814e6ef74954" (UID: "70ec3fd5-230c-4ce6-b093-814e6ef74954"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:40:31.168230 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:31.168200    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ec3fd5-230c-4ce6-b093-814e6ef74954-kube-api-access-d5s5s" pod "70ec3fd5-230c-4ce6-b093-814e6ef74954" (UID: "70ec3fd5-230c-4ce6-b093-814e6ef74954"). InnerVolumeSpecName "kube-api-access-d5s5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:31.265826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:31.265795    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5s5s\" (UniqueName: \"kubernetes.io/projected/70ec3fd5-230c-4ce6-b093-814e6ef74954-kube-api-access-d5s5s\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:31.265826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:31.265822    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:31.265826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:31.265832    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ec3fd5-230c-4ce6-b093-814e6ef74954-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:32.007244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:32.007146    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl" event={"ID":"70ec3fd5-230c-4ce6-b093-814e6ef74954","Type":"ContainerDied","Data":"b7e22d2bd4034cb2f0c445a6e8a50a29b0169d227a86b685becaca14379c479b"}
Nov 02 07:40:32.007531 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:32.007511    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7e22d2bd4034cb2f0c445a6e8a50a29b0169d227a86b685becaca14379c479b"
Nov 02 07:40:32.007797 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:32.007761    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"
Nov 02 07:40:32.010013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:32.009982    4898 generic.go:356] "Generic (PLEG): container finished" podID="90794722-dc5a-4d3c-9756-5584bb04d93c" containerID="c85f71663c920f103db8da6931395e779d1ee16442aa56d8af872def72fa0ec2" exitCode=0
Nov 02 07:40:32.010134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:32.010068    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76" event={"ID":"90794722-dc5a-4d3c-9756-5584bb04d93c","Type":"ContainerDied","Data":"c85f71663c920f103db8da6931395e779d1ee16442aa56d8af872def72fa0ec2"}
Nov 02 07:40:33.190781 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:33.190754    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:33.289329 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:33.289237    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90794722-dc5a-4d3c-9756-5584bb04d93c-kube-api-access-htrmt\" (UniqueName: \"kubernetes.io/projected/90794722-dc5a-4d3c-9756-5584bb04d93c-kube-api-access-htrmt\") pod \"90794722-dc5a-4d3c-9756-5584bb04d93c\" (UID: \"90794722-dc5a-4d3c-9756-5584bb04d93c\") "
Nov 02 07:40:33.289532 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:33.289393    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-xks-ca\") pod \"90794722-dc5a-4d3c-9756-5584bb04d93c\" (UID: \"90794722-dc5a-4d3c-9756-5584bb04d93c\") "
Nov 02 07:40:33.289532 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:33.289425    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-openshift-ca\") pod \"90794722-dc5a-4d3c-9756-5584bb04d93c\" (UID: \"90794722-dc5a-4d3c-9756-5584bb04d93c\") "
Nov 02 07:40:33.289913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:33.289856    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-openshift-ca" pod "90794722-dc5a-4d3c-9756-5584bb04d93c" (UID: "90794722-dc5a-4d3c-9756-5584bb04d93c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:33.291387 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:33.291359    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90794722-dc5a-4d3c-9756-5584bb04d93c-kube-api-access-htrmt" pod "90794722-dc5a-4d3c-9756-5584bb04d93c" (UID: "90794722-dc5a-4d3c-9756-5584bb04d93c"). InnerVolumeSpecName "kube-api-access-htrmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:33.291530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:33.291508    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-xks-ca" pod "90794722-dc5a-4d3c-9756-5584bb04d93c" (UID: "90794722-dc5a-4d3c-9756-5584bb04d93c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:40:33.390720 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:33.390690    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:33.390720 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:33.390718    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90794722-dc5a-4d3c-9756-5584bb04d93c-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:33.390928 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:33.390729    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htrmt\" (UniqueName: \"kubernetes.io/projected/90794722-dc5a-4d3c-9756-5584bb04d93c-kube-api-access-htrmt\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:34.013169 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.013132    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"]
Nov 02 07:40:34.013702 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.013689    4898 cpu_manager.go:414] "containerMap: removing container" podUID="90794722-dc5a-4d3c-9756-5584bb04d93c" containerName="revoke-keys"
Nov 02 07:40:34.013748 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.013705    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90794722-dc5a-4d3c-9756-5584bb04d93c" containerName="revoke-keys"
Nov 02 07:40:34.013748 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.013718    4898 cpu_manager.go:414] "containerMap: removing container" podUID="70ec3fd5-230c-4ce6-b093-814e6ef74954" containerName="revoke-keys"
Nov 02 07:40:34.013748 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.013724    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ec3fd5-230c-4ce6-b093-814e6ef74954" containerName="revoke-keys"
Nov 02 07:40:34.013863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.013853    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="90794722-dc5a-4d3c-9756-5584bb04d93c" containerName="revoke-keys"
Nov 02 07:40:34.013917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.013865    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ec3fd5-230c-4ce6-b093-814e6ef74954" containerName="revoke-keys"
Nov 02 07:40:34.017742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.017724    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:34.028295 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.028272    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"
Nov 02 07:40:34.028427 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.028316    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76" event={"ID":"90794722-dc5a-4d3c-9756-5584bb04d93c","Type":"ContainerDied","Data":"f1232e08927bec9f644071e0c8837e894b17f0955d1349d8fed56e0ed211be09"}
Nov 02 07:40:34.028427 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.028373    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1232e08927bec9f644071e0c8837e894b17f0955d1349d8fed56e0ed211be09"
Nov 02 07:40:34.036505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.036480    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"]
Nov 02 07:40:34.099593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.099549    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e40a3a13-nxb7b\" (UID: \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:34.099593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.099596    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn7kq\" (UniqueName: \"kubernetes.io/projected/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-kube-api-access-mn7kq\") pod \"maas-api-revoke-sub-e40a3a13-nxb7b\" (UID: \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:34.099989 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.099961    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e40a3a13-nxb7b\" (UID: \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:34.201370 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.201331    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e40a3a13-nxb7b\" (UID: \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:34.201857 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.201410    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e40a3a13-nxb7b\" (UID: \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:34.201857 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.201443    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7kq\" (UniqueName: \"kubernetes.io/projected/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-kube-api-access-mn7kq\") pod \"maas-api-revoke-sub-e40a3a13-nxb7b\" (UID: \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:34.202149 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.202126    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e40a3a13-nxb7b\" (UID: \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:34.203904 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.203869    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e40a3a13-nxb7b\" (UID: \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:34.224513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.224491    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7kq\" (UniqueName: \"kubernetes.io/projected/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-kube-api-access-mn7kq\") pod \"maas-api-revoke-sub-e40a3a13-nxb7b\" (UID: \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:34.333448 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.333368    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:34.643112 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:34.643080    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96fc6a9_7bc9_4a19_be83_b8fa78a650d0.slice/crio-915e304b6112b57e52560fabf34bf3d1751d12a0e98b3025071745c6159bc8b8 WatchSource:0}: Error finding container 915e304b6112b57e52560fabf34bf3d1751d12a0e98b3025071745c6159bc8b8: Status 404 returned error can't find the container with id 915e304b6112b57e52560fabf34bf3d1751d12a0e98b3025071745c6159bc8b8
Nov 02 07:40:34.645608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:34.645582    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"]
Nov 02 07:40:35.039387 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:35.039351    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b" event={"ID":"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0","Type":"ContainerStarted","Data":"c3ac452f7d0f0e7e667395b2ae13a6acecbb0e75d2a6d11e41ec40364f46a4de"}
Nov 02 07:40:35.039566 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:35.039411    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b" event={"ID":"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0","Type":"ContainerStarted","Data":"915e304b6112b57e52560fabf34bf3d1751d12a0e98b3025071745c6159bc8b8"}
Nov 02 07:40:35.088605 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:35.088528    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b" podStartSLOduration=2.088510653 podStartE2EDuration="2.088510653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:35.081588262 +0000 UTC m=+985.771915246" watchObservedRunningTime="2026-11-02 07:40:35.088510653 +0000 UTC m=+985.778837618"
Nov 02 07:40:36.049429 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:36.049390    4898 generic.go:356] "Generic (PLEG): container finished" podID="b96fc6a9-7bc9-4a19-be83-b8fa78a650d0" containerID="c3ac452f7d0f0e7e667395b2ae13a6acecbb0e75d2a6d11e41ec40364f46a4de" exitCode=0
Nov 02 07:40:36.049831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:36.049432    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b" event={"ID":"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0","Type":"ContainerDied","Data":"c3ac452f7d0f0e7e667395b2ae13a6acecbb0e75d2a6d11e41ec40364f46a4de"}
Nov 02 07:40:37.229040 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:37.229016    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:37.343361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:37.343206    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-xks-ca\") pod \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\" (UID: \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\") "
Nov 02 07:40:37.343361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:37.343280    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-kube-api-access-mn7kq\" (UniqueName: \"kubernetes.io/projected/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-kube-api-access-mn7kq\") pod \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\" (UID: \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\") "
Nov 02 07:40:37.343864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:37.343436    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-openshift-ca\") pod \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\" (UID: \"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0\") "
Nov 02 07:40:37.344767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:37.344734    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-openshift-ca" pod "b96fc6a9-7bc9-4a19-be83-b8fa78a650d0" (UID: "b96fc6a9-7bc9-4a19-be83-b8fa78a650d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:37.347221 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:37.347060    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-xks-ca" pod "b96fc6a9-7bc9-4a19-be83-b8fa78a650d0" (UID: "b96fc6a9-7bc9-4a19-be83-b8fa78a650d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:40:37.348358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:37.348327    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-kube-api-access-mn7kq" pod "b96fc6a9-7bc9-4a19-be83-b8fa78a650d0" (UID: "b96fc6a9-7bc9-4a19-be83-b8fa78a650d0"). InnerVolumeSpecName "kube-api-access-mn7kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:37.447506 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:37.447468    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:37.447506 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:37.447498    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:37.447506 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:37.447508    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn7kq\" (UniqueName: \"kubernetes.io/projected/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0-kube-api-access-mn7kq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:38.067599 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:38.067562    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"
Nov 02 07:40:38.067778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:38.067596    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b" event={"ID":"b96fc6a9-7bc9-4a19-be83-b8fa78a650d0","Type":"ContainerDied","Data":"915e304b6112b57e52560fabf34bf3d1751d12a0e98b3025071745c6159bc8b8"}
Nov 02 07:40:38.067778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:38.067647    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="915e304b6112b57e52560fabf34bf3d1751d12a0e98b3025071745c6159bc8b8"
Nov 02 07:40:39.440919 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:39.439317    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:40.384912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.379377    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 02 07:40:40.440912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.440485    4898 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:40.526912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.525039    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"]
Nov 02 07:40:40.551912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.544256    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b96fc6a9-7bc9-4a19-be83-b8fa78a650d0" containerName="revoke-keys"
Nov 02 07:40:40.551912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.544301    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96fc6a9-7bc9-4a19-be83-b8fa78a650d0" containerName="revoke-keys"
Nov 02 07:40:40.551912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.544571    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96fc6a9-7bc9-4a19-be83-b8fa78a650d0" containerName="revoke-keys"
Nov 02 07:40:40.765000 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.764356    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:40.798938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.798342    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:40.889559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.873649    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"]
Nov 02 07:40:40.889559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.873764    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:40.889559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.878901    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-77dc65fb-openshift-default-dockercfg-qp7lr\""
Nov 02 07:40:41.017601 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.974922    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-credential-socket\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.017601 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.974969    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-envoy\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.017601 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.975022    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f85612-ef12-4dad-826f-3711b7f72703-istio-podinfo\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.017601 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.975083    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f85612-ef12-4dad-826f-3711b7f72703-istiod-ca-cert\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.017601 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.975138    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-socket\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.017601 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.975242    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-istio-token\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.017601 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.975313    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-data\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.017601 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.975426    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-certs\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.017601 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:40.975454    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lx9g\" (UniqueName: \"kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-kube-api-access-6lx9g\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.112913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.111705    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.113538    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-socket\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.113683    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-istio-token\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.113768    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-data\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.113921    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-certs\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.113953    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lx9g\" (UniqueName: \"kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-kube-api-access-6lx9g\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.113981    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-credential-socket\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.114005    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-envoy\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.114066    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f85612-ef12-4dad-826f-3711b7f72703-istio-podinfo\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.114139    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f85612-ef12-4dad-826f-3711b7f72703-istiod-ca-cert\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.115013    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f85612-ef12-4dad-826f-3711b7f72703-istiod-ca-cert\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.115295    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-socket\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.115851    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-data\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.116146    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-certs\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.116629    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-credential-socket\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.122732    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"]
Nov 02 07:40:41.164011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.151129    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:41.176855 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.166238    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:41.180257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.179350    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:40:41.180257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.180206    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:40:41.195052 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.192010    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:40:41.195052 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.192981    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"]
Nov 02 07:40:41.195052 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.195009    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:40:41.282623 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.271341    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f85612-ef12-4dad-826f-3711b7f72703-istio-podinfo\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.282623 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.271974    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lx9g\" (UniqueName: \"kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-kube-api-access-6lx9g\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.283268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.283241    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d79ea55-x7b92\" (UID: \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:41.283507 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.283479    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d79ea55-x7b92\" (UID: \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:41.283637 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.283621    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxmjf\" (UniqueName: \"kubernetes.io/projected/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-kube-api-access-gxmjf\") pod \"maas-api-revoke-sub-4d79ea55-x7b92\" (UID: \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:41.292516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.292445    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-istio-token\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.292939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.292916    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-envoy\") pod \"e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") " pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.330911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.329933    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:40:41.331869 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.331730    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:41.375182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.371372    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:40:41.375182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.371436    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:41.423955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.403911    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:41.423955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.405138    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d79ea55-x7b92\" (UID: \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:41.423955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.405185    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d79ea55-x7b92\" (UID: \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:41.423955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.405210    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmjf\" (UniqueName: \"kubernetes.io/projected/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-kube-api-access-gxmjf\") pod \"maas-api-revoke-sub-4d79ea55-x7b92\" (UID: \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:41.423955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.405647    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:40:41.440945 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.429171    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d79ea55-x7b92\" (UID: \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:41.444178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.444143    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:41.454661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.454630    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d79ea55-x7b92\" (UID: \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:41.741171 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.724672    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmjf\" (UniqueName: \"kubernetes.io/projected/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-kube-api-access-gxmjf\") pod \"maas-api-revoke-sub-4d79ea55-x7b92\" (UID: \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:41.781375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.764795    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:42.031164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.025770    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:40:42.031164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.844292    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:40:42.031164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.027084    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:42.031164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.952820    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded"
Nov 02 07:40:42.031164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.892125    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:40:42.031164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:41.892384    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:42.032617 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.032598    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:40:42.041396 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.041355    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:42.112064 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.112009    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:42.216912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.213287    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:40:42.247915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.232210    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:42.339614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.339525    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:40:42.350979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.304052    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:40:42.351271 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.351228    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" podUID="534095d2-847d-4712-8f8d-acb19d774364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:42.362101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.362048    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:42.362739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.362696    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:42.369520 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.369498    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:40:42.444830 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.444789    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:42.455374 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.451351    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:40:42.474356 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.474326    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:40:42.808232 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.808106    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:40:42.811924 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.481216    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:42.962149 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.962053    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:40:42.977938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:42.962406    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp" podUID="3f4791d4-db06-4c5d-8423-745556f551a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:43.028911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.026743    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:40:43.029872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.029268    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:40:43.062921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.050932    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:43.062921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.051604    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:40:43.158402 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.157499    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:40:43.226348 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.205759    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:43.226348 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.205848    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:43.226348 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.205910    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:43.226348 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.221808    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:40:43.240920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.234604    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:40:43.356665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.349006    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-626zl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:40:43.356665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.349067    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" podUID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:40:43.358499 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.357015    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 02 07:40:43.416909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.407203    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:40:43.510068 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.509988    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:40:43.510308 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.510289    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"]
Nov 02 07:40:43.565991 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.565745    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" event={"ID":"f0f85612-ef12-4dad-826f-3711b7f72703","Type":"ContainerStarted","Data":"4634ee7585b3fdad536679629a52e8dce1d03e1a07592e77926032a9abfafa59"}
Nov 02 07:40:43.837620 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:43.837583    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"]
Nov 02 07:40:44.375127 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.375014    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"]
Nov 02 07:40:44.379578 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.379551    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:44.401942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.401348    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"]
Nov 02 07:40:44.478313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.478281    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f5e02d-pwkvj\" (UID: \"386bd924-0c5b-453a-aa47-8afd6bff971f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:44.478496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.478399    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f5e02d-pwkvj\" (UID: \"386bd924-0c5b-453a-aa47-8afd6bff971f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:44.478630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.478603    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9t2\" (UniqueName: \"kubernetes.io/projected/386bd924-0c5b-453a-aa47-8afd6bff971f-kube-api-access-hb9t2\") pod \"maas-api-revoke-sub-a3f5e02d-pwkvj\" (UID: \"386bd924-0c5b-453a-aa47-8afd6bff971f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:44.506333 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.506304    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"]
Nov 02 07:40:44.510746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.510719    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.534922 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.534773    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"]
Nov 02 07:40:44.546360 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.546330    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"]
Nov 02 07:40:44.551836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.551816    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.579778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.579740    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f5e02d-pwkvj\" (UID: \"386bd924-0c5b-453a-aa47-8afd6bff971f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:44.580003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.579856    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f5e02d-pwkvj\" (UID: \"386bd924-0c5b-453a-aa47-8afd6bff971f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:44.580084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.580063    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9t2\" (UniqueName: \"kubernetes.io/projected/386bd924-0c5b-453a-aa47-8afd6bff971f-kube-api-access-hb9t2\") pod \"maas-api-revoke-sub-a3f5e02d-pwkvj\" (UID: \"386bd924-0c5b-453a-aa47-8afd6bff971f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:44.580307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.580282    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92" event={"ID":"bb779ab5-b3d7-4bfa-a842-92b1ce6be750","Type":"ContainerStarted","Data":"ca5f19351a7358568c596e020db193907fc3d7b67b4b2b818d393fb90bdc821e"}
Nov 02 07:40:44.580370 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.580337    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92" event={"ID":"bb779ab5-b3d7-4bfa-a842-92b1ce6be750","Type":"ContainerStarted","Data":"b9d833d5a91307dc4c693cc0ddfca35fd5c43f98313d860a9700fc1999e8ba8e"}
Nov 02 07:40:44.581077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.581050    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"]
Nov 02 07:40:44.581230 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.581207    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f5e02d-pwkvj\" (UID: \"386bd924-0c5b-453a-aa47-8afd6bff971f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:44.583536 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.583511    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f5e02d-pwkvj\" (UID: \"386bd924-0c5b-453a-aa47-8afd6bff971f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:44.583690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.583574    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" event={"ID":"f0f85612-ef12-4dad-826f-3711b7f72703","Type":"ContainerStarted","Data":"229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5"}
Nov 02 07:40:44.603835 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.603813    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9t2\" (UniqueName: \"kubernetes.io/projected/386bd924-0c5b-453a-aa47-8afd6bff971f-kube-api-access-hb9t2\") pod \"maas-api-revoke-sub-a3f5e02d-pwkvj\" (UID: \"386bd924-0c5b-453a-aa47-8afd6bff971f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:44.649182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.649091    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92" podStartSLOduration=4.649078496 podStartE2EDuration="4.649078496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:44.646645732 +0000 UTC m=+995.336972696" watchObservedRunningTime="2026-11-02 07:40:44.649078496 +0000 UTC m=+995.339405460"
Nov 02 07:40:44.681717 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.681684    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8j4\" (UniqueName: \"kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-kube-api-access-vd8j4\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.681879 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.681719    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.681879 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.681759    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rbm\" (UniqueName: \"kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-kube-api-access-h2rbm\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.681994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.681931    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.682036 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.681989    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.682076 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682053    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.682112 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682087    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.682151 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682130    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.682201 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682173    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.682275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682258    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9683de3-4678-406d-bd92-5cad8ae2637b-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.682326 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682312    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.682431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682418    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.682494 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682479    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.682550 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682521    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.682689 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682664    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.682758 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682703    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.682883 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682864    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.682935 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.682914    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.694305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.694011    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" podStartSLOduration=4.693967347 podStartE2EDuration="4.693967347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:44.690826333 +0000 UTC m=+995.381153297" watchObservedRunningTime="2026-11-02 07:40:44.693967347 +0000 UTC m=+995.384294387"
Nov 02 07:40:44.700314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.700293    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786192    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8j4\" (UniqueName: \"kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-kube-api-access-vd8j4\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786241    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786279    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rbm\" (UniqueName: \"kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-kube-api-access-h2rbm\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786321    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786366    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786421    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786446    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786475    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786501    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786556    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9683de3-4678-406d-bd92-5cad8ae2637b-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786589    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786669    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786728    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786767    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786855    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786881    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.786986    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.787016    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.788910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.788684    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.794245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.792024    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.794245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.792679    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.794245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.792707    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.794245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.792954    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.794245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.793504    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.841307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.824278    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.841307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.832282    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.841307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.834289    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.841307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.837551    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9683de3-4678-406d-bd92-5cad8ae2637b-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.841307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.837777    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.844564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.844210    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.853271 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.853242    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.854043 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.854014    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.886976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.878991    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.888586 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.886874    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8j4\" (UniqueName: \"kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-kube-api-access-vd8j4\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.888729 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.888704    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rbm\" (UniqueName: \"kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-kube-api-access-h2rbm\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:44.889310 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:44.889285    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:44.992668 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:40:44.992621    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:40:45.007046 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:45.007015    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386bd924_0c5b_453a_aa47_8afd6bff971f.slice/crio-c0a469ad294020783d00b86e2c0a6ffb160cbdd46a65f44823451171f1541c2b WatchSource:0}: Error finding container c0a469ad294020783d00b86e2c0a6ffb160cbdd46a65f44823451171f1541c2b: Status 404 returned error can't find the container with id c0a469ad294020783d00b86e2c0a6ffb160cbdd46a65f44823451171f1541c2b
Nov 02 07:40:45.009997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.009974    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"]
Nov 02 07:40:45.124606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.124573    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:45.176045 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.176015    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:45.332615 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.332585    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:45.337386 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.337364    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:45.447083 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:45.447052    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9683de3_4678_406d_bd92_5cad8ae2637b.slice/crio-88c1aba8ec2e20821e43af3320a08a0cd68218b6c0901d576d4edc44b51c604c WatchSource:0}: Error finding container 88c1aba8ec2e20821e43af3320a08a0cd68218b6c0901d576d4edc44b51c604c: Status 404 returned error can't find the container with id 88c1aba8ec2e20821e43af3320a08a0cd68218b6c0901d576d4edc44b51c604c
Nov 02 07:40:45.449621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.449586    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:45.449697 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.449654    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:45.449697 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.449682    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:45.450218 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.450195    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"]
Nov 02 07:40:45.513041 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:45.513009    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4441a3e_ee8c_4af8_94f7_e9a4114ad603.slice/crio-87e33f3d490556bc529735d869a0311e56107ae0f6e96d21e2ad18d9349c462a WatchSource:0}: Error finding container 87e33f3d490556bc529735d869a0311e56107ae0f6e96d21e2ad18d9349c462a: Status 404 returned error can't find the container with id 87e33f3d490556bc529735d869a0311e56107ae0f6e96d21e2ad18d9349c462a
Nov 02 07:40:45.515406 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.515371    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:45.515510 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.515444    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:45.515510 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.515482    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:40:45.515675 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.515655    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"]
Nov 02 07:40:45.597011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.596984    4898 generic.go:356] "Generic (PLEG): container finished" podID="bb779ab5-b3d7-4bfa-a842-92b1ce6be750" containerID="ca5f19351a7358568c596e020db193907fc3d7b67b4b2b818d393fb90bdc821e" exitCode=0
Nov 02 07:40:45.597287 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.597212    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92" event={"ID":"bb779ab5-b3d7-4bfa-a842-92b1ce6be750","Type":"ContainerDied","Data":"ca5f19351a7358568c596e020db193907fc3d7b67b4b2b818d393fb90bdc821e"}
Nov 02 07:40:45.599691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.599667    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj" event={"ID":"386bd924-0c5b-453a-aa47-8afd6bff971f","Type":"ContainerStarted","Data":"3a79323281ba38d773dd6490149f56bfb334e94c4276ae1190030aa0bea894c1"}
Nov 02 07:40:45.599797 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.599713    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj" event={"ID":"386bd924-0c5b-453a-aa47-8afd6bff971f","Type":"ContainerStarted","Data":"c0a469ad294020783d00b86e2c0a6ffb160cbdd46a65f44823451171f1541c2b"}
Nov 02 07:40:45.602460 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.602434    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" event={"ID":"f9683de3-4678-406d-bd92-5cad8ae2637b","Type":"ContainerStarted","Data":"8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c"}
Nov 02 07:40:45.602570 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.602550    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" event={"ID":"f9683de3-4678-406d-bd92-5cad8ae2637b","Type":"ContainerStarted","Data":"88c1aba8ec2e20821e43af3320a08a0cd68218b6c0901d576d4edc44b51c604c"}
Nov 02 07:40:45.605918 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.605861    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" event={"ID":"b4441a3e-ee8c-4af8-94f7-e9a4114ad603","Type":"ContainerStarted","Data":"35729fa9dfe327ec7989e8008e873b3a48ad3cd09925adfc5836d62a30be3654"}
Nov 02 07:40:45.606005 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.605919    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" event={"ID":"b4441a3e-ee8c-4af8-94f7-e9a4114ad603","Type":"ContainerStarted","Data":"87e33f3d490556bc529735d869a0311e56107ae0f6e96d21e2ad18d9349c462a"}
Nov 02 07:40:45.606005 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.605949    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:45.607049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.607030    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:40:45.691007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.685063    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj" podStartSLOduration=1.6850458270000002 podStartE2EDuration="1.685045827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:45.679220028 +0000 UTC m=+996.369546993" watchObservedRunningTime="2026-11-02 07:40:45.685045827 +0000 UTC m=+996.375372792"
Nov 02 07:40:45.738824 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.738775    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" podStartSLOduration=1.7387611139999999 podStartE2EDuration="1.738761114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:45.734100909 +0000 UTC m=+996.424427873" watchObservedRunningTime="2026-11-02 07:40:45.738761114 +0000 UTC m=+996.429088078"
Nov 02 07:40:45.788853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:45.788736    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" podStartSLOduration=1.788717234 podStartE2EDuration="1.788717234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:45.783839233 +0000 UTC m=+996.474166223" watchObservedRunningTime="2026-11-02 07:40:45.788717234 +0000 UTC m=+996.479044199"
Nov 02 07:40:46.130547 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:46.130468    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:46.132012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:46.131982    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 07:40:46.132141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:46.132034    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" podUID="f9683de3-4678-406d-bd92-5cad8ae2637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 07:40:46.179912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:46.179693    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:46.179912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:46.179828    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 07:40:46.179912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:46.179872    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 07:40:46.618488 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:46.618434    4898 generic.go:356] "Generic (PLEG): container finished" podID="386bd924-0c5b-453a-aa47-8afd6bff971f" containerID="3a79323281ba38d773dd6490149f56bfb334e94c4276ae1190030aa0bea894c1" exitCode=0
Nov 02 07:40:46.618884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:46.618853    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj" event={"ID":"386bd924-0c5b-453a-aa47-8afd6bff971f","Type":"ContainerDied","Data":"3a79323281ba38d773dd6490149f56bfb334e94c4276ae1190030aa0bea894c1"}
Nov 02 07:40:46.924264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:46.924235    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:47.046282 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.046255    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-openshift-ca\") pod \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\" (UID: \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\") "
Nov 02 07:40:47.046435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.046316    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-kube-api-access-gxmjf\" (UniqueName: \"kubernetes.io/projected/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-kube-api-access-gxmjf\") pod \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\" (UID: \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\") "
Nov 02 07:40:47.046545 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.046531    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-xks-ca\") pod \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\" (UID: \"bb779ab5-b3d7-4bfa-a842-92b1ce6be750\") "
Nov 02 07:40:47.046735 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.046704    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-openshift-ca" pod "bb779ab5-b3d7-4bfa-a842-92b1ce6be750" (UID: "bb779ab5-b3d7-4bfa-a842-92b1ce6be750"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:47.047123 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.047096    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:47.048696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.048670    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-xks-ca" pod "bb779ab5-b3d7-4bfa-a842-92b1ce6be750" (UID: "bb779ab5-b3d7-4bfa-a842-92b1ce6be750"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:40:47.048790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.048717    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-kube-api-access-gxmjf" pod "bb779ab5-b3d7-4bfa-a842-92b1ce6be750" (UID: "bb779ab5-b3d7-4bfa-a842-92b1ce6be750"). InnerVolumeSpecName "kube-api-access-gxmjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:47.125715 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.125621    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 07:40:47.125715 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.125681    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" podUID="f9683de3-4678-406d-bd92-5cad8ae2637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 07:40:47.148843 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.148810    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:47.148843 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.148836    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxmjf\" (UniqueName: \"kubernetes.io/projected/bb779ab5-b3d7-4bfa-a842-92b1ce6be750-kube-api-access-gxmjf\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:47.181913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.181488    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 07:40:47.181913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.181555    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 07:40:47.629651 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.629620    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"
Nov 02 07:40:47.629795 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.629628    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92" event={"ID":"bb779ab5-b3d7-4bfa-a842-92b1ce6be750","Type":"ContainerDied","Data":"b9d833d5a91307dc4c693cc0ddfca35fd5c43f98313d860a9700fc1999e8ba8e"}
Nov 02 07:40:47.629795 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.629744    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9d833d5a91307dc4c693cc0ddfca35fd5c43f98313d860a9700fc1999e8ba8e"
Nov 02 07:40:47.938468 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:47.938444    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:48.125914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.125822    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-xks-ca\") pod \"386bd924-0c5b-453a-aa47-8afd6bff971f\" (UID: \"386bd924-0c5b-453a-aa47-8afd6bff971f\") "
Nov 02 07:40:48.126127 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.125939    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-openshift-ca\") pod \"386bd924-0c5b-453a-aa47-8afd6bff971f\" (UID: \"386bd924-0c5b-453a-aa47-8afd6bff971f\") "
Nov 02 07:40:48.136540 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.126261    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386bd924-0c5b-453a-aa47-8afd6bff971f-kube-api-access-hb9t2\" (UniqueName: \"kubernetes.io/projected/386bd924-0c5b-453a-aa47-8afd6bff971f-kube-api-access-hb9t2\") pod \"386bd924-0c5b-453a-aa47-8afd6bff971f\" (UID: \"386bd924-0c5b-453a-aa47-8afd6bff971f\") "
Nov 02 07:40:48.136540 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.128585    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 07:40:48.136540 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.128633    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" podUID="f9683de3-4678-406d-bd92-5cad8ae2637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 07:40:48.140321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.140214    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-xks-ca" pod "386bd924-0c5b-453a-aa47-8afd6bff971f" (UID: "386bd924-0c5b-453a-aa47-8afd6bff971f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:40:48.141325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.141296    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-openshift-ca" pod "386bd924-0c5b-453a-aa47-8afd6bff971f" (UID: "386bd924-0c5b-453a-aa47-8afd6bff971f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:48.141533 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.141512    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386bd924-0c5b-453a-aa47-8afd6bff971f-kube-api-access-hb9t2" pod "386bd924-0c5b-453a-aa47-8afd6bff971f" (UID: "386bd924-0c5b-453a-aa47-8afd6bff971f"). InnerVolumeSpecName "kube-api-access-hb9t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:48.178971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.178506    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 07:40:48.178971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.178563    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 07:40:48.228249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.228216    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb9t2\" (UniqueName: \"kubernetes.io/projected/386bd924-0c5b-453a-aa47-8afd6bff971f-kube-api-access-hb9t2\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:48.228444 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.228432    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:48.228510 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.228501    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386bd924-0c5b-453a-aa47-8afd6bff971f-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:48.645301 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.645261    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj" event={"ID":"386bd924-0c5b-453a-aa47-8afd6bff971f","Type":"ContainerDied","Data":"c0a469ad294020783d00b86e2c0a6ffb160cbdd46a65f44823451171f1541c2b"}
Nov 02 07:40:48.645483 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.645326    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0a469ad294020783d00b86e2c0a6ffb160cbdd46a65f44823451171f1541c2b"
Nov 02 07:40:48.645483 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:48.645392    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"
Nov 02 07:40:49.125282 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.125168    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 07:40:49.125282 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.125236    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" podUID="f9683de3-4678-406d-bd92-5cad8ae2637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 07:40:49.216033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.198008    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 07:40:49.216033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.198069    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 07:40:49.623750 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.623710    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:49.637555 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.637524    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:40:49.925161 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.925119    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"]
Nov 02 07:40:49.926417 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.926380    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bb779ab5-b3d7-4bfa-a842-92b1ce6be750" containerName="revoke-keys"
Nov 02 07:40:49.926584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.926568    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb779ab5-b3d7-4bfa-a842-92b1ce6be750" containerName="revoke-keys"
Nov 02 07:40:49.926710 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.926698    4898 cpu_manager.go:414] "containerMap: removing container" podUID="386bd924-0c5b-453a-aa47-8afd6bff971f" containerName="revoke-keys"
Nov 02 07:40:49.926812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.926799    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386bd924-0c5b-453a-aa47-8afd6bff971f" containerName="revoke-keys"
Nov 02 07:40:49.927083 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.927066    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb779ab5-b3d7-4bfa-a842-92b1ce6be750" containerName="revoke-keys"
Nov 02 07:40:49.927209 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.927197    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="386bd924-0c5b-453a-aa47-8afd6bff971f" containerName="revoke-keys"
Nov 02 07:40:49.931917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.931902    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"]
Nov 02 07:40:49.932021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.931969    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:49.935030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.933499    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7755j\" (UniqueName: \"kubernetes.io/projected/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-kube-api-access-7755j\") pod \"maas-api-revoke-sub-cfb47050-d8b6l\" (UID: \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:49.935030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.933544    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfb47050-d8b6l\" (UID: \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:49.935030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.933723    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfb47050-d8b6l\" (UID: \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:49.947332 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.947311    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:40:49.947447 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.947375    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:40:49.947447 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.947428    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:40:49.947447 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:49.947439    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:40:50.034664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:50.034615    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7755j\" (UniqueName: \"kubernetes.io/projected/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-kube-api-access-7755j\") pod \"maas-api-revoke-sub-cfb47050-d8b6l\" (UID: \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:50.034937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:50.034915    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfb47050-d8b6l\" (UID: \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:50.035235 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:50.035222    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfb47050-d8b6l\" (UID: \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:50.035666 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:50.035643    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfb47050-d8b6l\" (UID: \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:50.039921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:50.038294    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfb47050-d8b6l\" (UID: \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:50.101912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:50.096114    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7755j\" (UniqueName: \"kubernetes.io/projected/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-kube-api-access-7755j\") pod \"maas-api-revoke-sub-cfb47050-d8b6l\" (UID: \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:50.131925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:50.127291    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 07:40:50.131925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:50.127360    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" podUID="f9683de3-4678-406d-bd92-5cad8ae2637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 07:40:50.186010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:50.185240    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 07:40:50.186010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:50.185302    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 07:40:50.247358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:50.247326    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:50.654851 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:50.654826    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"]
Nov 02 07:40:50.712629 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:50.712441    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode909df00_b3ca_42cd_ac3a_1db5a9872fdd.slice/crio-e975b59daa63db482b18898f32b3c452c0ca90e4df1863791891519345016db2 WatchSource:0}: Error finding container e975b59daa63db482b18898f32b3c452c0ca90e4df1863791891519345016db2: Status 404 returned error can't find the container with id e975b59daa63db482b18898f32b3c452c0ca90e4df1863791891519345016db2
Nov 02 07:40:51.125498 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:51.125455    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 07:40:51.125649 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:51.125517    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" podUID="f9683de3-4678-406d-bd92-5cad8ae2637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 07:40:51.184912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:51.178881    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 07:40:51.184912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:51.178963    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 07:40:51.734965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:51.734919    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l" event={"ID":"e909df00-b3ca-42cd-ac3a-1db5a9872fdd","Type":"ContainerStarted","Data":"4d03392d7d624db0cf8fe9698e0ba20f898aa8e161ebff4d315d6503f0988bbb"}
Nov 02 07:40:51.735543 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:51.735513    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l" event={"ID":"e909df00-b3ca-42cd-ac3a-1db5a9872fdd","Type":"ContainerStarted","Data":"e975b59daa63db482b18898f32b3c452c0ca90e4df1863791891519345016db2"}
Nov 02 07:40:51.791517 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:51.789464    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l" podStartSLOduration=2.789443835 podStartE2EDuration="2.789443835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:40:51.776237278 +0000 UTC m=+1002.466564244" watchObservedRunningTime="2026-11-02 07:40:51.789443835 +0000 UTC m=+1002.479770800"
Nov 02 07:40:52.125775 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:52.125679    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 07:40:52.125775 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:52.125750    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" podUID="f9683de3-4678-406d-bd92-5cad8ae2637b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 07:40:52.177423 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:52.177390    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 02 07:40:52.177574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:52.177448    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 02 07:40:52.752120 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:52.752076    4898 generic.go:356] "Generic (PLEG): container finished" podID="e909df00-b3ca-42cd-ac3a-1db5a9872fdd" containerID="4d03392d7d624db0cf8fe9698e0ba20f898aa8e161ebff4d315d6503f0988bbb" exitCode=0
Nov 02 07:40:52.752643 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:52.752397    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l" event={"ID":"e909df00-b3ca-42cd-ac3a-1db5a9872fdd","Type":"ContainerDied","Data":"4d03392d7d624db0cf8fe9698e0ba20f898aa8e161ebff4d315d6503f0988bbb"}
Nov 02 07:40:53.221825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.221780    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:53.222397 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.222376    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:53.223415 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.223391    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 07:40:53.240746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.240717    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:53.241333 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.241312    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:53.242169 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.242144    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 07:40:53.950111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.950083    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:53.988617 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.988581    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-openshift-ca\") pod \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\" (UID: \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\") "
Nov 02 07:40:53.988863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.988631    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-kube-api-access-7755j\" (UniqueName: \"kubernetes.io/projected/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-kube-api-access-7755j\") pod \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\" (UID: \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\") "
Nov 02 07:40:53.988863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.988725    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-xks-ca\") pod \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\" (UID: \"e909df00-b3ca-42cd-ac3a-1db5a9872fdd\") "
Nov 02 07:40:53.989312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.989063    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-openshift-ca" pod "e909df00-b3ca-42cd-ac3a-1db5a9872fdd" (UID: "e909df00-b3ca-42cd-ac3a-1db5a9872fdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:40:53.989412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.989393    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:53.992624 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.992581    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-kube-api-access-7755j" pod "e909df00-b3ca-42cd-ac3a-1db5a9872fdd" (UID: "e909df00-b3ca-42cd-ac3a-1db5a9872fdd"). InnerVolumeSpecName "kube-api-access-7755j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:40:53.993010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:53.992982    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-xks-ca" pod "e909df00-b3ca-42cd-ac3a-1db5a9872fdd" (UID: "e909df00-b3ca-42cd-ac3a-1db5a9872fdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:40:54.090851 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:54.090748    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7755j\" (UniqueName: \"kubernetes.io/projected/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-kube-api-access-7755j\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:54.090851 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:54.090780    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e909df00-b3ca-42cd-ac3a-1db5a9872fdd-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:40:54.771179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:54.771152    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"
Nov 02 07:40:54.771355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:54.771184    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l" event={"ID":"e909df00-b3ca-42cd-ac3a-1db5a9872fdd","Type":"ContainerDied","Data":"e975b59daa63db482b18898f32b3c452c0ca90e4df1863791891519345016db2"}
Nov 02 07:40:54.771355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:54.771233    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e975b59daa63db482b18898f32b3c452c0ca90e4df1863791891519345016db2"
Nov 02 07:40:58.661678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.661632    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"]
Nov 02 07:40:58.662641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.662617    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e909df00-b3ca-42cd-ac3a-1db5a9872fdd" containerName="revoke-keys"
Nov 02 07:40:58.662641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.662643    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e909df00-b3ca-42cd-ac3a-1db5a9872fdd" containerName="revoke-keys"
Nov 02 07:40:58.662835 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.662820    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e909df00-b3ca-42cd-ac3a-1db5a9872fdd" containerName="revoke-keys"
Nov 02 07:40:58.667404 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.667382    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.699920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.697172    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 07:40:58.699920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.697257    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 07:40:58.699920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.697502    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bq7lr\""
Nov 02 07:40:58.699920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.697559    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 07:40:58.704271 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.704241    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"]
Nov 02 07:40:58.751915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.750367    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f75d9c81-bd5b-4263-b98d-b155f506c1f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.751915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.750413    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f75d9c81-bd5b-4263-b98d-b155f506c1f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.751915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.750653    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f75d9c81-bd5b-4263-b98d-b155f506c1f3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.751915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.750968    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f75d9c81-bd5b-4263-b98d-b155f506c1f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.751915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.751077    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75d9c81-bd5b-4263-b98d-b155f506c1f3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.751915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.751194    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-472j4\" (UniqueName: \"kubernetes.io/projected/f75d9c81-bd5b-4263-b98d-b155f506c1f3-kube-api-access-472j4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.751915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.751255    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f75d9c81-bd5b-4263-b98d-b155f506c1f3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.852905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.852852    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75d9c81-bd5b-4263-b98d-b155f506c1f3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.853102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.852981    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-472j4\" (UniqueName: \"kubernetes.io/projected/f75d9c81-bd5b-4263-b98d-b155f506c1f3-kube-api-access-472j4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.853102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.853033    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f75d9c81-bd5b-4263-b98d-b155f506c1f3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.853102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.853097    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f75d9c81-bd5b-4263-b98d-b155f506c1f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.853260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.853141    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f75d9c81-bd5b-4263-b98d-b155f506c1f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.853260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.853204    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f75d9c81-bd5b-4263-b98d-b155f506c1f3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.853378 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.853308    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f75d9c81-bd5b-4263-b98d-b155f506c1f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.853901 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.853854    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f75d9c81-bd5b-4263-b98d-b155f506c1f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.855749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.855721    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75d9c81-bd5b-4263-b98d-b155f506c1f3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.855849 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.855771    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f75d9c81-bd5b-4263-b98d-b155f506c1f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.856154 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.856133    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f75d9c81-bd5b-4263-b98d-b155f506c1f3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.856477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.856458    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f75d9c81-bd5b-4263-b98d-b155f506c1f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.876353 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.876326    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-472j4\" (UniqueName: \"kubernetes.io/projected/f75d9c81-bd5b-4263-b98d-b155f506c1f3-kube-api-access-472j4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.876445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.876397    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f75d9c81-bd5b-4263-b98d-b155f506c1f3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bdrxw\" (UID: \"f75d9c81-bd5b-4263-b98d-b155f506c1f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:58.991909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:58.991801    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:40:59.202298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.202260    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"]
Nov 02 07:40:59.207048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.207027    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:40:59.222402 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.222278    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:40:59.222402 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.222371    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:40:59.226477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.226454    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"]
Nov 02 07:40:59.269334 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.269244    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:40:59.269651 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.269631    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:40:59.363604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.362935    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4c5c758-xk7hc\" (UID: \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:40:59.363604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.363055    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4c5c758-xk7hc\" (UID: \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:40:59.363604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.363106    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghqbc\" (UniqueName: \"kubernetes.io/projected/54c3d3a8-727e-403e-ac99-3841f6b43ad0-kube-api-access-ghqbc\") pod \"maas-api-revoke-sub-e4c5c758-xk7hc\" (UID: \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:40:59.364033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.363777    4898 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 07:40:59.367779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.367748    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"]
Nov 02 07:40:59.454490 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.454456    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"]
Nov 02 07:40:59.464772 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.464737    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4c5c758-xk7hc\" (UID: \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:40:59.464967 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.464842    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4c5c758-xk7hc\" (UID: \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:40:59.464967 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.464899    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqbc\" (UniqueName: \"kubernetes.io/projected/54c3d3a8-727e-403e-ac99-3841f6b43ad0-kube-api-access-ghqbc\") pod \"maas-api-revoke-sub-e4c5c758-xk7hc\" (UID: \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:40:59.465134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.465114    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:40:59.465570 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.465543    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4c5c758-xk7hc\" (UID: \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:40:59.467743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.467721    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4c5c758-xk7hc\" (UID: \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:40:59.489661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.489627    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"]
Nov 02 07:40:59.492098 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.492059    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghqbc\" (UniqueName: \"kubernetes.io/projected/54c3d3a8-727e-403e-ac99-3841f6b43ad0-kube-api-access-ghqbc\") pod \"maas-api-revoke-sub-e4c5c758-xk7hc\" (UID: \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:40:59.521756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.521678    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:40:59.567289 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.566779    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f856dfed-q9vk7\" (UID: \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:40:59.567289 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.566986    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9qmt\" (UniqueName: \"kubernetes.io/projected/ec2df4ed-fb73-4014-995e-eddd01ae1c18-kube-api-access-t9qmt\") pod \"maas-api-revoke-sub-f856dfed-q9vk7\" (UID: \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:40:59.567289 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.567091    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f856dfed-q9vk7\" (UID: \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:40:59.678198 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.669780    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qmt\" (UniqueName: \"kubernetes.io/projected/ec2df4ed-fb73-4014-995e-eddd01ae1c18-kube-api-access-t9qmt\") pod \"maas-api-revoke-sub-f856dfed-q9vk7\" (UID: \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:40:59.678198 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.669918    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f856dfed-q9vk7\" (UID: \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:40:59.678198 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.670053    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f856dfed-q9vk7\" (UID: \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:40:59.685174 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.680368    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f856dfed-q9vk7\" (UID: \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:40:59.687530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.686406    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f856dfed-q9vk7\" (UID: \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:40:59.705662 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.705545    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qmt\" (UniqueName: \"kubernetes.io/projected/ec2df4ed-fb73-4014-995e-eddd01ae1c18-kube-api-access-t9qmt\") pod \"maas-api-revoke-sub-f856dfed-q9vk7\" (UID: \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:40:59.791494 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.791400    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:40:59.824178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.824143    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw" event={"ID":"f75d9c81-bd5b-4263-b98d-b155f506c1f3","Type":"ContainerStarted","Data":"e6ec1f93ab0ee4ccb3209cd771a42bee0a6394ba3152588c561b93f1b32e406c"}
Nov 02 07:40:59.885533 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:40:59.874882    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"]
Nov 02 07:40:59.885533 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:40:59.875769    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c3d3a8_727e_403e_ac99_3841f6b43ad0.slice/crio-c620b443815a577feb74a4a829970960cdf6635f330276129a48306e6d1bbf28 WatchSource:0}: Error finding container c620b443815a577feb74a4a829970960cdf6635f330276129a48306e6d1bbf28: Status 404 returned error can't find the container with id c620b443815a577feb74a4a829970960cdf6635f330276129a48306e6d1bbf28
Nov 02 07:41:00.106984 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:41:00.106943    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2df4ed_fb73_4014_995e_eddd01ae1c18.slice/crio-d70f086fe3cbc0f6c3b203be88a5eacd03b0e2a73ef4434c6c1926f3889fb555 WatchSource:0}: Error finding container d70f086fe3cbc0f6c3b203be88a5eacd03b0e2a73ef4434c6c1926f3889fb555: Status 404 returned error can't find the container with id d70f086fe3cbc0f6c3b203be88a5eacd03b0e2a73ef4434c6c1926f3889fb555
Nov 02 07:41:00.110342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:00.110319    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"]
Nov 02 07:41:00.912918 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:00.912299    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc" event={"ID":"54c3d3a8-727e-403e-ac99-3841f6b43ad0","Type":"ContainerStarted","Data":"c13a9909cacfb600e554f12145d240345b0a91434c440ffad8e17bcc44e96460"}
Nov 02 07:41:00.912918 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:00.912375    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc" event={"ID":"54c3d3a8-727e-403e-ac99-3841f6b43ad0","Type":"ContainerStarted","Data":"c620b443815a577feb74a4a829970960cdf6635f330276129a48306e6d1bbf28"}
Nov 02 07:41:00.931181 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:00.931145    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7" event={"ID":"ec2df4ed-fb73-4014-995e-eddd01ae1c18","Type":"ContainerStarted","Data":"0996037d27715c806d8a6550a6b6385bdd12e78d1bd85f05338001c781460d56"}
Nov 02 07:41:00.931432 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:00.931408    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7" event={"ID":"ec2df4ed-fb73-4014-995e-eddd01ae1c18","Type":"ContainerStarted","Data":"d70f086fe3cbc0f6c3b203be88a5eacd03b0e2a73ef4434c6c1926f3889fb555"}
Nov 02 07:41:01.011936 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:00.993391    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc" podStartSLOduration=1.993371476 podStartE2EDuration="1.993371476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:00.960322013 +0000 UTC m=+1011.650649002" watchObservedRunningTime="2026-11-02 07:41:00.993371476 +0000 UTC m=+1011.683698440"
Nov 02 07:41:01.383684 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:01.370370    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7" podStartSLOduration=2.370345976 podStartE2EDuration="2.370345976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:01.026426269 +0000 UTC m=+1011.716753236" watchObservedRunningTime="2026-11-02 07:41:01.370345976 +0000 UTC m=+1012.060672942"
Nov 02 07:41:02.216912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:02.191594    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:41:02.328210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:02.202469    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:02.328210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:02.204221    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 07:41:02.328210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:02.319302    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 07:41:02.363912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:02.342521    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:02.364133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:02.364027    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:02.534968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:02.520238    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:02.673927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:02.670294    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:02.708990 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:02.708925    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:02.825966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:02.819559    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:03.049977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.040912    4898 generic.go:356] "Generic (PLEG): container finished" podID="54c3d3a8-727e-403e-ac99-3841f6b43ad0" containerID="c13a9909cacfb600e554f12145d240345b0a91434c440ffad8e17bcc44e96460" exitCode=0
Nov 02 07:41:03.049977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.041189    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc" event={"ID":"54c3d3a8-727e-403e-ac99-3841f6b43ad0","Type":"ContainerDied","Data":"c13a9909cacfb600e554f12145d240345b0a91434c440ffad8e17bcc44e96460"}
Nov 02 07:41:03.149974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.148077    4898 generic.go:356] "Generic (PLEG): container finished" podID="ec2df4ed-fb73-4014-995e-eddd01ae1c18" containerID="0996037d27715c806d8a6550a6b6385bdd12e78d1bd85f05338001c781460d56" exitCode=0
Nov 02 07:41:03.149974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.148356    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7" event={"ID":"ec2df4ed-fb73-4014-995e-eddd01ae1c18","Type":"ContainerDied","Data":"0996037d27715c806d8a6550a6b6385bdd12e78d1bd85f05338001c781460d56"}
Nov 02 07:41:03.399067 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.399013    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:41:03.400059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.399979    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 07:41:03.400553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.400529    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 02 07:41:03.400760 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.400719    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:03.401369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.401351    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:41:03.401580 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.401563    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:41:03.401717 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.401706    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:41:03.401837 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.401827    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:41:03.419102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.419069    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:41:03.424176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.420948    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:41:03.521505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.521451    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 07:41:03.709966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.691459    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Nov 02 07:41:03.757707 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.757659    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 02 07:41:03.841915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:03.823123    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:04.075344 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.075248    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:41:04.075629 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.075612    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:41:04.075807 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.075792    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:41:04.075990 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.075976    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:41:04.428073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.428027    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:04.513915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.460366    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:04.513915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.487180    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:41:04.769944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.710270    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 07:41:04.769944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.757664    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 07:41:04.824490 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.458133    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:04.825601 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.825459    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:04.860820 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.858867    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:04.885031 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:04.884981    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 02 07:41:05.207952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.033308    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:41:05.207952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.187213    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:41:05.331399 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.331322    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:41:05.376453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.345815    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:41:05.376453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.347356    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded"
Nov 02 07:41:05.420595 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.418855    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:41:05.442213 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.429525    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:41:05.442213 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.433127    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:05.443172 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.442954    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:41:05.447163 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.447139    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:41:05.462469 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.462308    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:41:05.479384 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.479358    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:41:05.483011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.482986    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"
Nov 02 07:41:05.483156 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.483136    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7" event={"ID":"ec2df4ed-fb73-4014-995e-eddd01ae1c18","Type":"ContainerDied","Data":"d70f086fe3cbc0f6c3b203be88a5eacd03b0e2a73ef4434c6c1926f3889fb555"}
Nov 02 07:41:05.483219 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.483174    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d70f086fe3cbc0f6c3b203be88a5eacd03b0e2a73ef4434c6c1926f3889fb555"
Nov 02 07:41:05.486450 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.486432    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:41:05.598926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.598825    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-xks-ca\") pod \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\" (UID: \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\") "
Nov 02 07:41:05.599237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.599213    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-openshift-ca\") pod \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\" (UID: \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\") "
Nov 02 07:41:05.599797 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.599776    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-xks-ca\") pod \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\" (UID: \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\") "
Nov 02 07:41:05.599977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.599957    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-openshift-ca\") pod \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\" (UID: \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\") "
Nov 02 07:41:05.600165 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.600145    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2df4ed-fb73-4014-995e-eddd01ae1c18-kube-api-access-t9qmt\" (UniqueName: \"kubernetes.io/projected/ec2df4ed-fb73-4014-995e-eddd01ae1c18-kube-api-access-t9qmt\") pod \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\" (UID: \"ec2df4ed-fb73-4014-995e-eddd01ae1c18\") "
Nov 02 07:41:05.600439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.600419    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c3d3a8-727e-403e-ac99-3841f6b43ad0-kube-api-access-ghqbc\" (UniqueName: \"kubernetes.io/projected/54c3d3a8-727e-403e-ac99-3841f6b43ad0-kube-api-access-ghqbc\") pod \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\" (UID: \"54c3d3a8-727e-403e-ac99-3841f6b43ad0\") "
Nov 02 07:41:05.601732 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.599691    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-openshift-ca" pod "54c3d3a8-727e-403e-ac99-3841f6b43ad0" (UID: "54c3d3a8-727e-403e-ac99-3841f6b43ad0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:41:05.601996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.600704    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-openshift-ca" pod "ec2df4ed-fb73-4014-995e-eddd01ae1c18" (UID: "ec2df4ed-fb73-4014-995e-eddd01ae1c18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:41:05.602390 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.601869    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-xks-ca" pod "54c3d3a8-727e-403e-ac99-3841f6b43ad0" (UID: "54c3d3a8-727e-403e-ac99-3841f6b43ad0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:41:05.603443 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.603413    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-xks-ca" pod "ec2df4ed-fb73-4014-995e-eddd01ae1c18" (UID: "ec2df4ed-fb73-4014-995e-eddd01ae1c18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:41:05.603970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.603868    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c3d3a8-727e-403e-ac99-3841f6b43ad0-kube-api-access-ghqbc" pod "54c3d3a8-727e-403e-ac99-3841f6b43ad0" (UID: "54c3d3a8-727e-403e-ac99-3841f6b43ad0"). InnerVolumeSpecName "kube-api-access-ghqbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:41:05.604270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.604221    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2df4ed-fb73-4014-995e-eddd01ae1c18-kube-api-access-t9qmt" pod "ec2df4ed-fb73-4014-995e-eddd01ae1c18" (UID: "ec2df4ed-fb73-4014-995e-eddd01ae1c18"). InnerVolumeSpecName "kube-api-access-t9qmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:41:05.703030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.702989    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9qmt\" (UniqueName: \"kubernetes.io/projected/ec2df4ed-fb73-4014-995e-eddd01ae1c18-kube-api-access-t9qmt\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:41:05.703030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.703028    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghqbc\" (UniqueName: \"kubernetes.io/projected/54c3d3a8-727e-403e-ac99-3841f6b43ad0-kube-api-access-ghqbc\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:41:05.703265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.703045    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:41:05.703265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.703060    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c3d3a8-727e-403e-ac99-3841f6b43ad0-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:41:05.703265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.703074    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:41:05.703265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:05.703088    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec2df4ed-fb73-4014-995e-eddd01ae1c18-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:41:06.504952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:06.503033    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc" event={"ID":"54c3d3a8-727e-403e-ac99-3841f6b43ad0","Type":"ContainerDied","Data":"c620b443815a577feb74a4a829970960cdf6635f330276129a48306e6d1bbf28"}
Nov 02 07:41:06.504952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:06.503062    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"
Nov 02 07:41:06.504952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:06.503093    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c620b443815a577feb74a4a829970960cdf6635f330276129a48306e6d1bbf28"
Nov 02 07:41:07.327846 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:07.327807    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:07.327962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:07.327874    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:07.515422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:07.515380    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw" event={"ID":"f75d9c81-bd5b-4263-b98d-b155f506c1f3","Type":"ContainerStarted","Data":"76b71606c269d658b348c5347f92ad99ec2e799d1ddf4cf8985843be257756cc"}
Nov 02 07:41:07.515812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:07.515467    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:41:07.589566 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:07.589463    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw" podStartSLOduration=1.627049811 podStartE2EDuration="9.589448148s" totalImagesPullingTime="7.962398337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:07.574366165 +0000 UTC m=+1018.264693131" watchObservedRunningTime="2026-11-02 07:41:07.589448148 +0000 UTC m=+1018.279775113"
Nov 02 07:41:08.527960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:08.527932    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bdrxw"
Nov 02 07:41:09.647073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:09.647046    4898 scope.go:122] "RemoveContainer" containerID="f6182d5f82b05e791fd16ffad4bbbd47eed5d6a85f292b0faf289f240ede13ec"
Nov 02 07:41:09.657042 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:09.657019    4898 scope.go:122] "RemoveContainer" containerID="cd7feeb9a09237fb5e45580d00c7b7fa0b82fac1b4096ce507bd4ffaf84ade44"
Nov 02 07:41:09.667278 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:09.667257    4898 scope.go:122] "RemoveContainer" containerID="6e5c5ff508ee19d984056b1572b924fce2614853bba37c67d69baa50ea2fd4dd"
Nov 02 07:41:09.677711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:09.677691    4898 scope.go:122] "RemoveContainer" containerID="5141b5f628131396d02bf7acd7e65f2dc800b599940850ce07203b5fc18c4c65"
Nov 02 07:41:14.989486 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:41:14.989451    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:41:16.556331 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.556295    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"]
Nov 02 07:41:16.557246 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.557224    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ec2df4ed-fb73-4014-995e-eddd01ae1c18" containerName="revoke-keys"
Nov 02 07:41:16.557391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.557247    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2df4ed-fb73-4014-995e-eddd01ae1c18" containerName="revoke-keys"
Nov 02 07:41:16.557391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.557299    4898 cpu_manager.go:414] "containerMap: removing container" podUID="54c3d3a8-727e-403e-ac99-3841f6b43ad0" containerName="revoke-keys"
Nov 02 07:41:16.557391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.557307    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c3d3a8-727e-403e-ac99-3841f6b43ad0" containerName="revoke-keys"
Nov 02 07:41:16.557552 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.557468    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c3d3a8-727e-403e-ac99-3841f6b43ad0" containerName="revoke-keys"
Nov 02 07:41:16.557552 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.557481    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2df4ed-fb73-4014-995e-eddd01ae1c18" containerName="revoke-keys"
Nov 02 07:41:16.563848 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.563826    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"
Nov 02 07:41:16.579860 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.579830    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"]
Nov 02 07:41:16.584661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.584636    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fe085e\""
Nov 02 07:41:16.584960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.584940    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fe085e-dockercfg-97xt8\""
Nov 02 07:41:16.659496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.659446    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"]
Nov 02 07:41:16.666811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.666782    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"
Nov 02 07:41:16.669752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.669720    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83278155-b02a-4944-ae0b-6d3c215c33a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8\" (UID: \"83278155-b02a-4944-ae0b-6d3c215c33a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"
Nov 02 07:41:16.769537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.769504    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"]
Nov 02 07:41:16.778103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.777756    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt6zw\" (UniqueName: \"kubernetes.io/projected/83278155-b02a-4944-ae0b-6d3c215c33a7-kube-api-access-mt6zw\") pod \"payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8\" (UID: \"83278155-b02a-4944-ae0b-6d3c215c33a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"
Nov 02 07:41:16.879426 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.879338    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m\" (UID: \"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"
Nov 02 07:41:16.879593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.879529    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt6zw\" (UniqueName: \"kubernetes.io/projected/83278155-b02a-4944-ae0b-6d3c215c33a7-kube-api-access-mt6zw\") pod \"payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8\" (UID: \"83278155-b02a-4944-ae0b-6d3c215c33a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"
Nov 02 07:41:16.879667 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.879596    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83278155-b02a-4944-ae0b-6d3c215c33a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8\" (UID: \"83278155-b02a-4944-ae0b-6d3c215c33a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"
Nov 02 07:41:16.879727 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.879678    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffjn\" (UniqueName: \"kubernetes.io/projected/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-kube-api-access-cffjn\") pod \"payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m\" (UID: \"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"
Nov 02 07:41:16.881033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.881006    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83278155-b02a-4944-ae0b-6d3c215c33a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8\" (UID: \"83278155-b02a-4944-ae0b-6d3c215c33a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"
Nov 02 07:41:16.904185 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.904148    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt6zw\" (UniqueName: \"kubernetes.io/projected/83278155-b02a-4944-ae0b-6d3c215c33a7-kube-api-access-mt6zw\") pod \"payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8\" (UID: \"83278155-b02a-4944-ae0b-6d3c215c33a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"
Nov 02 07:41:16.980838 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.980799    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffjn\" (UniqueName: \"kubernetes.io/projected/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-kube-api-access-cffjn\") pod \"payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m\" (UID: \"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"
Nov 02 07:41:16.981020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.981006    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m\" (UID: \"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"
Nov 02 07:41:16.982015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:16.981988    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m\" (UID: \"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"
Nov 02 07:41:17.004924 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:17.004861    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffjn\" (UniqueName: \"kubernetes.io/projected/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-kube-api-access-cffjn\") pod \"payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m\" (UID: \"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"
Nov 02 07:41:17.091061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:17.091021    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"
Nov 02 07:41:17.184025 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:17.183720    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"
Nov 02 07:41:17.401523 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:17.400130    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"]
Nov 02 07:41:17.404210 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:41:17.404183    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a40ebf4_6e54_498e_b5bd_837dcea6b6f5.slice/crio-14e757ae363dee0a97a68d2e820b1d45c948459a3606ce1b043e7ad36fa89dda WatchSource:0}: Error finding container 14e757ae363dee0a97a68d2e820b1d45c948459a3606ce1b043e7ad36fa89dda: Status 404 returned error can't find the container with id 14e757ae363dee0a97a68d2e820b1d45c948459a3606ce1b043e7ad36fa89dda
Nov 02 07:41:17.546407 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:41:17.546363    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83278155_b02a_4944_ae0b_6d3c215c33a7.slice/crio-247b8ae1e44f632e6da0bd462e5efd52b01e18108977ebe0fb7fe76e912db0bc WatchSource:0}: Error finding container 247b8ae1e44f632e6da0bd462e5efd52b01e18108977ebe0fb7fe76e912db0bc: Status 404 returned error can't find the container with id 247b8ae1e44f632e6da0bd462e5efd52b01e18108977ebe0fb7fe76e912db0bc
Nov 02 07:41:17.553809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:17.549649    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"]
Nov 02 07:41:17.647359 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:17.647321    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8" event={"ID":"83278155-b02a-4944-ae0b-6d3c215c33a7","Type":"ContainerStarted","Data":"247b8ae1e44f632e6da0bd462e5efd52b01e18108977ebe0fb7fe76e912db0bc"}
Nov 02 07:41:17.649210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:17.649181    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m" event={"ID":"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5","Type":"ContainerStarted","Data":"c30a18352b08055329425a821880d0b05003f6888c3e4ddc32daf1c97e265a32"}
Nov 02 07:41:17.649325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:17.649235    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m" event={"ID":"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5","Type":"ContainerStarted","Data":"14e757ae363dee0a97a68d2e820b1d45c948459a3606ce1b043e7ad36fa89dda"}
Nov 02 07:41:17.649325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:17.649279    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"
Nov 02 07:41:17.695034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:17.694726    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m" podStartSLOduration=1.694706362 podStartE2EDuration="1.694706362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:17.691515551 +0000 UTC m=+1028.381842516" watchObservedRunningTime="2026-11-02 07:41:17.694706362 +0000 UTC m=+1028.385033327"
Nov 02 07:41:18.659111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:18.659055    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8" event={"ID":"83278155-b02a-4944-ae0b-6d3c215c33a7","Type":"ContainerStarted","Data":"ac84b145294d35fb64bde47b319597c6a32a3a81adc51cf3f30668beb5700640"}
Nov 02 07:41:18.660085 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:18.659228    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"
Nov 02 07:41:18.701991 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:18.701940    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8" podStartSLOduration=2.7019263970000003 podStartE2EDuration="2.701926397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:18.698934907 +0000 UTC m=+1029.389261872" watchObservedRunningTime="2026-11-02 07:41:18.701926397 +0000 UTC m=+1029.392253360"
Nov 02 07:41:24.338142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:24.336991    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:24.339337 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:24.338686    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 02 07:41:24.514913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:24.486567    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:24.573916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:24.551602    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:41:24.736046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:24.735991    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:24.774039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:24.773989    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:24.841919 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:24.838989    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:24.870916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:24.869974    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:25.260915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.259981    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded"
Nov 02 07:41:25.366916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.359367    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 02 07:41:25.366916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.359313    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:25.366916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.363116    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:41:25.366916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.363199    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:41:25.537996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.537876    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 07:41:25.539846 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.539818    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:41:25.561163 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.561116    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:25.562696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.562660    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 02 07:41:25.563940 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.563919    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:41:25.761196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.760024    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:25.796087 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.795031    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 07:41:25.819914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.813714    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:41:25.819914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.813776    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:41:25.912944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.907484    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:25.912944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.907917    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:41:25.926926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.917380    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Nov 02 07:41:25.970795 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:25.969457    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:41:26.369063 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:26.369015    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:26.371498 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:26.370179    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:41:26.547272 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:26.547216    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:26.606917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:26.571569    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:26.699123 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:26.604738    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:26.699686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:26.587089    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:26.700194 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:26.700168    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:41:26.775996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:26.775946    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 07:41:26.897913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:26.895271    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 07:41:26.932654 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:26.927223    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:27.083142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:27.083042    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Nov 02 07:41:27.515949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:27.469102    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:27.597139 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:27.597086    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:41:27.598579 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:27.598533    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Nov 02 07:41:27.719911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:27.718911    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 02 07:41:27.941693 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:27.941650    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:41:27.942532 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:27.942500    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 07:41:27.943394 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:27.943371    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded"
Nov 02 07:41:27.944121 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:27.944096    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:28.015912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:28.015848    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 02 07:41:28.090965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:28.086989    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:28.090965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:28.089442    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 07:41:28.516925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:28.516243    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:28.675442 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:28.675347    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:28.680494 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:28.680120    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"
Nov 02 07:41:29.024993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:29.021250    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:29.024993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:29.023717    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:29.024993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:29.024447    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:41:29.024993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:29.024723    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 07:41:29.037913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:29.033602    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:29.086927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:29.084004    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:29.087589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:29.087535    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:29.093918 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:29.092995    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 07:41:29.590937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:29.582593    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:29.697131 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:29.697056    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:29.733521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:29.733486    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:41:29.733845 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:29.733825    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"
Nov 02 07:41:30.043915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.024702    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:41:30.043915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.043834    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 07:41:30.046951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.046911    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:30.071935 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.062917    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 02 07:41:30.161916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.137011    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:30.264914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.164437    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:41:30.264914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.191513    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 07:41:30.633913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.633304    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnq5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:41:30.633913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.633398    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" podUID="dea9d7e7-103a-4e9a-8824-e62a72191cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:30.695908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.694942    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 02 07:41:30.726066 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.725997    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:30.728928 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.726090    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:41:30.751803 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.738018    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:41:30.764641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.764611    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:41:30.769466 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.769435    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:41:30.842245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:30.842213    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:41:31.025559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:31.025510    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:41:31.102695 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:31.102652    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 02 07:41:31.109521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:31.109492    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:41:31.135879 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:31.135848    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:41:36.853874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:36.853834    4898 generic.go:356] "Generic (PLEG): container finished" podID="f0f85612-ef12-4dad-826f-3711b7f72703" containerID="229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5" exitCode=0
Nov 02 07:41:36.854448 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:36.853923    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" event={"ID":"f0f85612-ef12-4dad-826f-3711b7f72703","Type":"ContainerDied","Data":"229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5"}
Nov 02 07:41:36.854513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:36.854500    4898 scope.go:122] "RemoveContainer" containerID="229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5"
Nov 02 07:41:36.856177 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:36.856143    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:36.856305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:36.856233    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:36.856305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:36.856281    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:37.888964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:37.888922    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" event={"ID":"f0f85612-ef12-4dad-826f-3711b7f72703","Type":"ContainerStarted","Data":"344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca"}
Nov 02 07:41:38.332690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:38.332648    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:41:38.332973 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:38.332949    4898 patch_prober.go:29] interesting pod/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 07:41:38.333038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:38.333002    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 07:41:39.332813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:39.332775    4898 patch_prober.go:29] interesting pod/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 07:41:39.333259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:39.332829    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 07:41:40.332772 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:40.332740    4898 patch_prober.go:29] interesting pod/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 07:41:40.332963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:40.332791    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 07:41:41.332881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:41.332846    4898 patch_prober.go:29] interesting pod/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 07:41:41.333105 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:41.332910    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 07:41:42.332553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:42.332511    4898 patch_prober.go:29] interesting pod/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 07:41:42.332769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:42.332585    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 07:41:43.338060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:43.338023    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:41:43.338540 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:43.338465    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:41:43.339103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:43.339081    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:41:44.992049 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:41:44.992011    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:41:52.023282 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.023242    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"]
Nov 02 07:41:52.029448 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.029416    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.051229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.051192    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.051395 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.051244    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.051454 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.051400    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.051454 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.051427    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.051529 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.051513    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.051608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.051585    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.051754 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.051659    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.051754 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.051684    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqt4r\" (UniqueName: \"kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-kube-api-access-rqt4r\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.051754 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.051737    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.079909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.066274    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"]
Nov 02 07:41:52.153184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.153149    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.153374 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.153221    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.153374 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.153268    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.153374 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.153293    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqt4r\" (UniqueName: \"kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-kube-api-access-rqt4r\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.153374 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.153340    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.153590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.153549    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.153649 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.153594    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.153736 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.153698    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.153736 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.153726    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.153849 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.153754    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.153849 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.153764    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.154027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.153922    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.154155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.154132    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.154381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.154348    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.155738 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.155711    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.156269 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.156247    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.191328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.191292    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.194595 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.194563    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqt4r\" (UniqueName: \"kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-kube-api-access-rqt4r\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.345930 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.345824    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:52.666765 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:41:52.666735    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb10a74e_3c67_4331_9cc8_c54ce4a948ba.slice/crio-dd750b3c638b8d6432bf5399fd2b656319fe3c0b1f45a3b1f7f00bcd56181a5c WatchSource:0}: Error finding container dd750b3c638b8d6432bf5399fd2b656319fe3c0b1f45a3b1f7f00bcd56181a5c: Status 404 returned error can't find the container with id dd750b3c638b8d6432bf5399fd2b656319fe3c0b1f45a3b1f7f00bcd56181a5c
Nov 02 07:41:52.669575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.669535    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:52.669678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.669627    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:52.669678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.669671    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:41:52.672260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:52.672181    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"]
Nov 02 07:41:53.144506 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:53.144421    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" event={"ID":"cb10a74e-3c67-4331-9cc8-c54ce4a948ba","Type":"ContainerStarted","Data":"46007fc9cf13af4ecf908433ea2442364f666219ff66dc5b674dea286731ab0c"}
Nov 02 07:41:53.144506 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:53.144486    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" event={"ID":"cb10a74e-3c67-4331-9cc8-c54ce4a948ba","Type":"ContainerStarted","Data":"dd750b3c638b8d6432bf5399fd2b656319fe3c0b1f45a3b1f7f00bcd56181a5c"}
Nov 02 07:41:53.192546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:53.192477    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" podStartSLOduration=2.192452007 podStartE2EDuration="2.192452007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:53.189475144 +0000 UTC m=+1063.879802109" watchObservedRunningTime="2026-11-02 07:41:53.192452007 +0000 UTC m=+1063.882778975"
Nov 02 07:41:53.346665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:53.346631    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:41:53.348368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:53.348346    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 02 07:41:53.348428 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:53.348400    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" podUID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 02 07:41:54.347246 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:54.347215    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 02 07:41:54.347672 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:54.347309    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" podUID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 02 07:41:55.346459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:55.346419    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 02 07:41:55.346639 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:55.346480    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" podUID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 02 07:41:56.288342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.288299    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"]
Nov 02 07:41:56.293523 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.293496    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:41:56.302084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.302053    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"]
Nov 02 07:41:56.322651 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.322613    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:41:56.322878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.322720    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:41:56.322878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.322765    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:41:56.322878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.322792    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:41:56.344976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.344936    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddba9d62-g6shf\" (UID: \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:41:56.345166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.345049    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swtcb\" (UniqueName: \"kubernetes.io/projected/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-kube-api-access-swtcb\") pod \"maas-api-revoke-sub-ddba9d62-g6shf\" (UID: \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:41:56.345166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.345126    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddba9d62-g6shf\" (UID: \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:41:56.346775 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.346738    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 02 07:41:56.346938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.346798    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" podUID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 02 07:41:56.446686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.446651    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddba9d62-g6shf\" (UID: \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:41:56.446862 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.446756    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddba9d62-g6shf\" (UID: \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:41:56.446862 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.446836    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swtcb\" (UniqueName: \"kubernetes.io/projected/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-kube-api-access-swtcb\") pod \"maas-api-revoke-sub-ddba9d62-g6shf\" (UID: \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:41:56.447603 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.447573    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddba9d62-g6shf\" (UID: \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:41:56.449085 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.449066    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddba9d62-g6shf\" (UID: \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:41:56.486840 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.486801    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swtcb\" (UniqueName: \"kubernetes.io/projected/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-kube-api-access-swtcb\") pod \"maas-api-revoke-sub-ddba9d62-g6shf\" (UID: \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:41:56.611927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.611826    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:41:56.944965 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:41:56.944933    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91ed93d_6a2c_40e2_9bc8_b46de087e9e4.slice/crio-2e10310b056c5928dd27054be7d1f3bd8d6aa5e63387595afd2ffacfb66c7c29 WatchSource:0}: Error finding container 2e10310b056c5928dd27054be7d1f3bd8d6aa5e63387595afd2ffacfb66c7c29: Status 404 returned error can't find the container with id 2e10310b056c5928dd27054be7d1f3bd8d6aa5e63387595afd2ffacfb66c7c29
Nov 02 07:41:56.949779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:56.949753    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"]
Nov 02 07:41:57.043160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:57.043123    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"]
Nov 02 07:41:57.048114 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:57.048088    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f377b3a2-h255n"]
Nov 02 07:41:57.229647 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:57.229561    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf" event={"ID":"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4","Type":"ContainerStarted","Data":"2e10310b056c5928dd27054be7d1f3bd8d6aa5e63387595afd2ffacfb66c7c29"}
Nov 02 07:41:57.346477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:57.346436    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 02 07:41:57.346846 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:57.346502    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" podUID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 02 07:41:57.501748 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:57.501708    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08592b61-ba78-4240-8436-91ac9458863c" path="/var/lib/kubelet/pods/08592b61-ba78-4240-8436-91ac9458863c/volumes"
Nov 02 07:41:58.197501 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.197093    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"]
Nov 02 07:41:58.203101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.202544    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"
Nov 02 07:41:58.204008 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.203980    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"]
Nov 02 07:41:58.218303 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.218274    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cc2bcd-dockercfg-xbrhf\""
Nov 02 07:41:58.230692 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.230669    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cc2bcd\""
Nov 02 07:41:58.289913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.272358    4898 generic.go:356] "Generic (PLEG): container finished" podID="e91ed93d-6a2c-40e2-9bc8-b46de087e9e4" containerID="64c2d9d6cba14a333927931e5857d5a56d74629021b098c07aeda98510af10ea" exitCode=0
Nov 02 07:41:58.289913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.273224    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf" event={"ID":"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4","Type":"ContainerDied","Data":"64c2d9d6cba14a333927931e5857d5a56d74629021b098c07aeda98510af10ea"}
Nov 02 07:41:58.298009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.297920    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8\" (UID: \"d3ae1d71-59c1-4344-ad51-f0e815a60c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"
Nov 02 07:41:58.321914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.319192    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4tv5\" (UniqueName: \"kubernetes.io/projected/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-kube-api-access-z4tv5\") pod \"payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8\" (UID: \"d3ae1d71-59c1-4344-ad51-f0e815a60c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"
Nov 02 07:41:58.384911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.355023    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 02 07:41:58.384911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.358964    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" podUID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 02 07:41:58.508988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.499392    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8\" (UID: \"d3ae1d71-59c1-4344-ad51-f0e815a60c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"
Nov 02 07:41:58.508988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.499771    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4tv5\" (UniqueName: \"kubernetes.io/projected/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-kube-api-access-z4tv5\") pod \"payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8\" (UID: \"d3ae1d71-59c1-4344-ad51-f0e815a60c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"
Nov 02 07:41:58.562915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.535598    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8\" (UID: \"d3ae1d71-59c1-4344-ad51-f0e815a60c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"
Nov 02 07:41:58.587546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.586999    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4tv5\" (UniqueName: \"kubernetes.io/projected/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-kube-api-access-z4tv5\") pod \"payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8\" (UID: \"d3ae1d71-59c1-4344-ad51-f0e815a60c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"
Nov 02 07:41:58.703107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:58.703068    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"
Nov 02 07:41:59.034749 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:41:59.034704    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ae1d71_59c1_4344_ad51_f0e815a60c9c.slice/crio-b2c5a2a2ce600f7ecd1360d64c4f343504d8c7c2625681d674b29aa24b52f9eb WatchSource:0}: Error finding container b2c5a2a2ce600f7ecd1360d64c4f343504d8c7c2625681d674b29aa24b52f9eb: Status 404 returned error can't find the container with id b2c5a2a2ce600f7ecd1360d64c4f343504d8c7c2625681d674b29aa24b52f9eb
Nov 02 07:41:59.039576 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.038754    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"]
Nov 02 07:41:59.286188 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.286076    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8" event={"ID":"d3ae1d71-59c1-4344-ad51-f0e815a60c9c","Type":"ContainerStarted","Data":"1a8fc3790b1048734e68e0bef35b6bfbf086b4ac6b77630a4618f0c1270af624"}
Nov 02 07:41:59.286188 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.286149    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8" event={"ID":"d3ae1d71-59c1-4344-ad51-f0e815a60c9c","Type":"ContainerStarted","Data":"b2c5a2a2ce600f7ecd1360d64c4f343504d8c7c2625681d674b29aa24b52f9eb"}
Nov 02 07:41:59.286188 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.286187    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"
Nov 02 07:41:59.328967 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.328907    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8" podStartSLOduration=1.328865173 podStartE2EDuration="1.328865173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:41:59.327667184 +0000 UTC m=+1070.017994149" watchObservedRunningTime="2026-11-02 07:41:59.328865173 +0000 UTC m=+1070.019192137"
Nov 02 07:41:59.346813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.346773    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 02 07:41:59.347027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.346840    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" podUID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 02 07:41:59.432471 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.432439    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:41:59.623503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.623414    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-xks-ca\") pod \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\" (UID: \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\") "
Nov 02 07:41:59.623503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.623491    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-kube-api-access-swtcb\" (UniqueName: \"kubernetes.io/projected/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-kube-api-access-swtcb\") pod \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\" (UID: \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\") "
Nov 02 07:41:59.623816 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.623541    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-openshift-ca\") pod \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\" (UID: \"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4\") "
Nov 02 07:41:59.624391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.624354    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-openshift-ca" pod "e91ed93d-6a2c-40e2-9bc8-b46de087e9e4" (UID: "e91ed93d-6a2c-40e2-9bc8-b46de087e9e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:41:59.625339 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.625317    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:41:59.626764 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.626720    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-kube-api-access-swtcb" pod "e91ed93d-6a2c-40e2-9bc8-b46de087e9e4" (UID: "e91ed93d-6a2c-40e2-9bc8-b46de087e9e4"). InnerVolumeSpecName "kube-api-access-swtcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:41:59.629313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.629284    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-xks-ca" pod "e91ed93d-6a2c-40e2-9bc8-b46de087e9e4" (UID: "e91ed93d-6a2c-40e2-9bc8-b46de087e9e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:41:59.727039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.727000    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:41:59.727039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:41:59.727036    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swtcb\" (UniqueName: \"kubernetes.io/projected/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4-kube-api-access-swtcb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:42:00.301906 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:00.301856    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf" event={"ID":"e91ed93d-6a2c-40e2-9bc8-b46de087e9e4","Type":"ContainerDied","Data":"2e10310b056c5928dd27054be7d1f3bd8d6aa5e63387595afd2ffacfb66c7c29"}
Nov 02 07:42:00.302095 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:00.301941    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e10310b056c5928dd27054be7d1f3bd8d6aa5e63387595afd2ffacfb66c7c29"
Nov 02 07:42:00.302095 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:00.301957    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"
Nov 02 07:42:00.405484 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:00.405448    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:42:00.406963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:00.406939    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:42:00.407881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:00.407854    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:42:01.385369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.385333    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"]
Nov 02 07:42:01.386072 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.386053    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e91ed93d-6a2c-40e2-9bc8-b46de087e9e4" containerName="revoke-keys"
Nov 02 07:42:01.386072 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.386071    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e91ed93d-6a2c-40e2-9bc8-b46de087e9e4" containerName="revoke-keys"
Nov 02 07:42:01.386224 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.386169    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e91ed93d-6a2c-40e2-9bc8-b46de087e9e4" containerName="revoke-keys"
Nov 02 07:42:01.390855 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.390403    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.406872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.406844    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b3e5f7-openshift-default-dockercfg-vdqkq\""
Nov 02 07:42:01.426247 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.426214    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"]
Nov 02 07:42:01.567918 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.564832    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdk44\" (UniqueName: \"kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-kube-api-access-fdk44\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.579913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.579013    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-token\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.579913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.579099    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-socket\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.579913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.579166    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-credential-socket\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.579913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.579211    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-certs\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.579913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.579261    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-data\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.579913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.579340    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-podinfo\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.579913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.579414    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-envoy\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.579913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.579446    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82360126-e2f7-4bf0-9abf-3c82168fb797-istiod-ca-cert\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.684250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.683946    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdk44\" (UniqueName: \"kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-kube-api-access-fdk44\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.684250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.684001    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-token\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.684250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.684052    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-socket\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.684250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.684107    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-credential-socket\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.684250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.684154    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-certs\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.684250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.684201    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-data\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.684250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.684256    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-podinfo\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.684770 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.684318    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-envoy\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.684770 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.684348    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82360126-e2f7-4bf0-9abf-3c82168fb797-istiod-ca-cert\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.685256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.685225    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82360126-e2f7-4bf0-9abf-3c82168fb797-istiod-ca-cert\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.686070 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.686042    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-socket\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.686306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.686287    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-credential-socket\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.686555 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.686510    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-certs\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.686815 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.686731    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-data\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.689645 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.689616    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-envoy\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.690409 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.690385    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-podinfo\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.709149 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.709120    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-token\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.709505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.709478    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdk44\" (UniqueName: \"kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-kube-api-access-fdk44\") pod \"e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") " pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:01.713098 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:01.713070    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:02.039002 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:42:02.038964    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82360126_e2f7_4bf0_9abf_3c82168fb797.slice/crio-9ca4716df53190b364898171748864998fc5effb1c94e10dd73476a3a801256b WatchSource:0}: Error finding container 9ca4716df53190b364898171748864998fc5effb1c94e10dd73476a3a801256b: Status 404 returned error can't find the container with id 9ca4716df53190b364898171748864998fc5effb1c94e10dd73476a3a801256b
Nov 02 07:42:02.041993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:02.041950    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:02.042105 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:02.042042    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:02.042105 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:02.042091    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:02.043669 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:02.043646    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"]
Nov 02 07:42:02.326505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:02.326416    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" event={"ID":"82360126-e2f7-4bf0-9abf-3c82168fb797","Type":"ContainerStarted","Data":"eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834"}
Nov 02 07:42:02.326505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:02.326485    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" event={"ID":"82360126-e2f7-4bf0-9abf-3c82168fb797","Type":"ContainerStarted","Data":"9ca4716df53190b364898171748864998fc5effb1c94e10dd73476a3a801256b"}
Nov 02 07:42:02.374033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:02.373971    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" podStartSLOduration=1.373952298 podStartE2EDuration="1.373952298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:42:02.372363517 +0000 UTC m=+1073.062690509" watchObservedRunningTime="2026-11-02 07:42:02.373952298 +0000 UTC m=+1073.064279263"
Nov 02 07:42:02.714083 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:02.714045    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:03.720048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:03.720015    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:04.344590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:04.344545    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:04.345652 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:04.345626    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:06.970691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:06.970659    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"]
Nov 02 07:42:06.975426 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:06.975406    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"
Nov 02 07:42:06.990357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:06.990329    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b3e5f7\""
Nov 02 07:42:07.002800 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:07.002764    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b3e5f7-dockercfg-grs7p\""
Nov 02 07:42:07.018186 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:07.018148    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"]
Nov 02 07:42:07.119058 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:07.107256    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz6gq\" (UniqueName: \"kubernetes.io/projected/2383994e-608b-4b83-b8fa-2a24992bc937-kube-api-access-rz6gq\") pod \"payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm\" (UID: \"2383994e-608b-4b83-b8fa-2a24992bc937\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"
Nov 02 07:42:07.119058 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:07.107519    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2383994e-608b-4b83-b8fa-2a24992bc937-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm\" (UID: \"2383994e-608b-4b83-b8fa-2a24992bc937\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"
Nov 02 07:42:07.211603 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:07.210250    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2383994e-608b-4b83-b8fa-2a24992bc937-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm\" (UID: \"2383994e-608b-4b83-b8fa-2a24992bc937\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"
Nov 02 07:42:07.211603 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:07.210537    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz6gq\" (UniqueName: \"kubernetes.io/projected/2383994e-608b-4b83-b8fa-2a24992bc937-kube-api-access-rz6gq\") pod \"payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm\" (UID: \"2383994e-608b-4b83-b8fa-2a24992bc937\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"
Nov 02 07:42:07.211859 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:07.211834    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2383994e-608b-4b83-b8fa-2a24992bc937-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm\" (UID: \"2383994e-608b-4b83-b8fa-2a24992bc937\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"
Nov 02 07:42:07.270998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:07.270748    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz6gq\" (UniqueName: \"kubernetes.io/projected/2383994e-608b-4b83-b8fa-2a24992bc937-kube-api-access-rz6gq\") pod \"payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm\" (UID: \"2383994e-608b-4b83-b8fa-2a24992bc937\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"
Nov 02 07:42:07.291390 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:07.291241    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"
Nov 02 07:42:07.603174 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:42:07.603137    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2383994e_608b_4b83_b8fa_2a24992bc937.slice/crio-0c303de889439abae1f0d4c73f509ceeac755c41ce82abdafef0acb29255e5c0 WatchSource:0}: Error finding container 0c303de889439abae1f0d4c73f509ceeac755c41ce82abdafef0acb29255e5c0: Status 404 returned error can't find the container with id 0c303de889439abae1f0d4c73f509ceeac755c41ce82abdafef0acb29255e5c0
Nov 02 07:42:07.607249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:07.607222    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"]
Nov 02 07:42:08.387960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:08.387921    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm" event={"ID":"2383994e-608b-4b83-b8fa-2a24992bc937","Type":"ContainerStarted","Data":"3fe5450c90beaa2f56ca79b9ecd93570cc8cd0034226197e17472ea31ffa2034"}
Nov 02 07:42:08.388398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:08.387981    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm" event={"ID":"2383994e-608b-4b83-b8fa-2a24992bc937","Type":"ContainerStarted","Data":"0c303de889439abae1f0d4c73f509ceeac755c41ce82abdafef0acb29255e5c0"}
Nov 02 07:42:08.388398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:08.388021    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"
Nov 02 07:42:08.430376 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:08.430316    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm" podStartSLOduration=2.430301238 podStartE2EDuration="2.430301238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:42:08.428771264 +0000 UTC m=+1079.119098229" watchObservedRunningTime="2026-11-02 07:42:08.430301238 +0000 UTC m=+1079.120628202"
Nov 02 07:42:10.305561 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:10.305528    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"
Nov 02 07:42:12.050914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:12.045659    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"]
Nov 02 07:42:12.060912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:12.057105    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada4a680-gf8g7"]
Nov 02 07:42:13.466528 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:13.466491    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb098e81-4393-4754-a778-2a8234a3c035" path="/var/lib/kubelet/pods/fb098e81-4393-4754-a778-2a8234a3c035/volumes"
Nov 02 07:42:14.990956 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:42:14.990913    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:42:19.404650 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:19.404622    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"
Nov 02 07:42:22.040110 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:22.040069    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"]
Nov 02 07:42:22.052369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:22.052331    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50829d69-55xvj"]
Nov 02 07:42:23.467639 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:23.467603    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e26c7f4f-7a4b-4d82-acc7-6335c1f221d9" path="/var/lib/kubelet/pods/e26c7f4f-7a4b-4d82-acc7-6335c1f221d9/volumes"
Nov 02 07:42:25.047917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:25.045348    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"]
Nov 02 07:42:25.053496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:25.053460    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c85fd61a-d87rj"]
Nov 02 07:42:25.466850 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:25.466811    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a40a65b-52bc-425c-903f-410b8f0c784a" path="/var/lib/kubelet/pods/3a40a65b-52bc-425c-903f-410b8f0c784a/volumes"
Nov 02 07:42:40.035617 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.035583    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"]
Nov 02 07:42:40.047393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.047366    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edbb1d83-vmsqs"]
Nov 02 07:42:40.647757 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.647724    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"]
Nov 02 07:42:40.653602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.653583    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:40.681658 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.681632    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:42:40.681979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.681961    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:42:40.682139 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.682126    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:42:40.682250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.682239    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:42:40.703588 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.703562    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"]
Nov 02 07:42:40.839666 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.839625    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w74n7\" (UniqueName: \"kubernetes.io/projected/bcea2473-5168-41f4-8d71-7e2136764f1b-kube-api-access-w74n7\") pod \"maas-api-revoke-sub-0d204bad-s27ql\" (UID: \"bcea2473-5168-41f4-8d71-7e2136764f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:40.839915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.839677    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d204bad-s27ql\" (UID: \"bcea2473-5168-41f4-8d71-7e2136764f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:40.839915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.839823    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d204bad-s27ql\" (UID: \"bcea2473-5168-41f4-8d71-7e2136764f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:40.941563 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.941528    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w74n7\" (UniqueName: \"kubernetes.io/projected/bcea2473-5168-41f4-8d71-7e2136764f1b-kube-api-access-w74n7\") pod \"maas-api-revoke-sub-0d204bad-s27ql\" (UID: \"bcea2473-5168-41f4-8d71-7e2136764f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:40.941716 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.941672    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d204bad-s27ql\" (UID: \"bcea2473-5168-41f4-8d71-7e2136764f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:40.941827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.941812    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d204bad-s27ql\" (UID: \"bcea2473-5168-41f4-8d71-7e2136764f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:40.942492 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.942469    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d204bad-s27ql\" (UID: \"bcea2473-5168-41f4-8d71-7e2136764f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:40.944255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.944233    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d204bad-s27ql\" (UID: \"bcea2473-5168-41f4-8d71-7e2136764f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:40.965112 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:40.965088    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w74n7\" (UniqueName: \"kubernetes.io/projected/bcea2473-5168-41f4-8d71-7e2136764f1b-kube-api-access-w74n7\") pod \"maas-api-revoke-sub-0d204bad-s27ql\" (UID: \"bcea2473-5168-41f4-8d71-7e2136764f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:41.004646 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:41.004612    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:41.327742 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:42:41.327710    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcea2473_5168_41f4_8d71_7e2136764f1b.slice/crio-cc73af5e2b1e0590f80ba01a7281386fde808b996ee26e5d621b0a64b5c6322a WatchSource:0}: Error finding container cc73af5e2b1e0590f80ba01a7281386fde808b996ee26e5d621b0a64b5c6322a: Status 404 returned error can't find the container with id cc73af5e2b1e0590f80ba01a7281386fde808b996ee26e5d621b0a64b5c6322a
Nov 02 07:42:41.333430 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:41.333376    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"]
Nov 02 07:42:41.467931 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:41.467879    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddcf02e0-410d-4ac7-8214-50da19a83fbb" path="/var/lib/kubelet/pods/ddcf02e0-410d-4ac7-8214-50da19a83fbb/volumes"
Nov 02 07:42:41.961778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:41.961744    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql" event={"ID":"bcea2473-5168-41f4-8d71-7e2136764f1b","Type":"ContainerStarted","Data":"cb10b9515c6333fe197df997aa39699d8cbb493fa07036ef0791547e8968fb35"}
Nov 02 07:42:41.961951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:41.961805    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql" event={"ID":"bcea2473-5168-41f4-8d71-7e2136764f1b","Type":"ContainerStarted","Data":"cc73af5e2b1e0590f80ba01a7281386fde808b996ee26e5d621b0a64b5c6322a"}
Nov 02 07:42:42.008333 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:42.008278    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql" podStartSLOduration=2.008263364 podStartE2EDuration="2.008263364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:42:42.003788041 +0000 UTC m=+1112.694115005" watchObservedRunningTime="2026-11-02 07:42:42.008263364 +0000 UTC m=+1112.698590375"
Nov 02 07:42:42.974458 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:42.974418    4898 generic.go:356] "Generic (PLEG): container finished" podID="bcea2473-5168-41f4-8d71-7e2136764f1b" containerID="cb10b9515c6333fe197df997aa39699d8cbb493fa07036ef0791547e8968fb35" exitCode=0
Nov 02 07:42:42.974884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:42.974500    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql" event={"ID":"bcea2473-5168-41f4-8d71-7e2136764f1b","Type":"ContainerDied","Data":"cb10b9515c6333fe197df997aa39699d8cbb493fa07036ef0791547e8968fb35"}
Nov 02 07:42:43.039082 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:43.039043    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"]
Nov 02 07:42:43.044152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:43.044118    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba23cf6d-dk6vl"]
Nov 02 07:42:43.470830 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:43.470788    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dbbd712-ab89-4aee-8d0b-fb15e56e6c00" path="/var/lib/kubelet/pods/7dbbd712-ab89-4aee-8d0b-fb15e56e6c00/volumes"
Nov 02 07:42:44.258826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:44.258761    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:44.353325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:44.353283    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-openshift-ca\") pod \"bcea2473-5168-41f4-8d71-7e2136764f1b\" (UID: \"bcea2473-5168-41f4-8d71-7e2136764f1b\") "
Nov 02 07:42:44.353519 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:44.353435    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-xks-ca\") pod \"bcea2473-5168-41f4-8d71-7e2136764f1b\" (UID: \"bcea2473-5168-41f4-8d71-7e2136764f1b\") "
Nov 02 07:42:44.353519 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:44.353497    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcea2473-5168-41f4-8d71-7e2136764f1b-kube-api-access-w74n7\" (UniqueName: \"kubernetes.io/projected/bcea2473-5168-41f4-8d71-7e2136764f1b-kube-api-access-w74n7\") pod \"bcea2473-5168-41f4-8d71-7e2136764f1b\" (UID: \"bcea2473-5168-41f4-8d71-7e2136764f1b\") "
Nov 02 07:42:44.353731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:44.353706    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-openshift-ca" pod "bcea2473-5168-41f4-8d71-7e2136764f1b" (UID: "bcea2473-5168-41f4-8d71-7e2136764f1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:42:44.354197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:44.354175    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:42:44.357193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:44.357167    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-xks-ca" pod "bcea2473-5168-41f4-8d71-7e2136764f1b" (UID: "bcea2473-5168-41f4-8d71-7e2136764f1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:42:44.357336 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:44.357316    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcea2473-5168-41f4-8d71-7e2136764f1b-kube-api-access-w74n7" pod "bcea2473-5168-41f4-8d71-7e2136764f1b" (UID: "bcea2473-5168-41f4-8d71-7e2136764f1b"). InnerVolumeSpecName "kube-api-access-w74n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:42:44.455248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:44.455217    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcea2473-5168-41f4-8d71-7e2136764f1b-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:42:44.455248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:44.455243    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w74n7\" (UniqueName: \"kubernetes.io/projected/bcea2473-5168-41f4-8d71-7e2136764f1b-kube-api-access-w74n7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:42:45.038608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:45.035292    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql" event={"ID":"bcea2473-5168-41f4-8d71-7e2136764f1b","Type":"ContainerDied","Data":"cc73af5e2b1e0590f80ba01a7281386fde808b996ee26e5d621b0a64b5c6322a"}
Nov 02 07:42:45.038608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:45.035365    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc73af5e2b1e0590f80ba01a7281386fde808b996ee26e5d621b0a64b5c6322a"
Nov 02 07:42:45.038608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:45.035459    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"
Nov 02 07:42:45.125689 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:42:45.124538    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:42:46.105582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:46.105551    4898 generic.go:356] "Generic (PLEG): container finished" podID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerID="eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834" exitCode=0
Nov 02 07:42:46.106248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:46.106218    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" event={"ID":"82360126-e2f7-4bf0-9abf-3c82168fb797","Type":"ContainerDied","Data":"eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834"}
Nov 02 07:42:46.106984 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:46.106958    4898 scope.go:122] "RemoveContainer" containerID="eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834"
Nov 02 07:42:46.111106 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:46.111058    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:46.111228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:46.111161    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:46.111293 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:46.111234    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:42:47.118325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:47.118291    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" event={"ID":"82360126-e2f7-4bf0-9abf-3c82168fb797","Type":"ContainerStarted","Data":"3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9"}
Nov 02 07:42:47.713691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:47.713652    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:47.715242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:47.715215    4898 patch_prober.go:29] interesting pod/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 07:42:47.715379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:47.715276    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 07:42:48.421941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.421910    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"]
Nov 02 07:42:48.422600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.422587    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bcea2473-5168-41f4-8d71-7e2136764f1b" containerName="revoke-keys"
Nov 02 07:42:48.422650 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.422601    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcea2473-5168-41f4-8d71-7e2136764f1b" containerName="revoke-keys"
Nov 02 07:42:48.422709 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.422700    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcea2473-5168-41f4-8d71-7e2136764f1b" containerName="revoke-keys"
Nov 02 07:42:48.426604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.426587    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:48.441911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.441860    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:42:48.447670 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.447642    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"]
Nov 02 07:42:48.451691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.451666    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:42:48.451846 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.451725    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:42:48.451846 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.451726    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:42:48.489474 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.489410    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lr4\" (UniqueName: \"kubernetes.io/projected/96ad00cf-432f-4d56-add3-7330ca4bfa10-kube-api-access-p9lr4\") pod \"maas-api-revoke-sub-db66d052-cxrpn\" (UID: \"96ad00cf-432f-4d56-add3-7330ca4bfa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:48.489649 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.489537    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db66d052-cxrpn\" (UID: \"96ad00cf-432f-4d56-add3-7330ca4bfa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:48.489709 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.489685    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db66d052-cxrpn\" (UID: \"96ad00cf-432f-4d56-add3-7330ca4bfa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:48.594981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.594931    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db66d052-cxrpn\" (UID: \"96ad00cf-432f-4d56-add3-7330ca4bfa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:48.595444 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.595412    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db66d052-cxrpn\" (UID: \"96ad00cf-432f-4d56-add3-7330ca4bfa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:48.595951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.595923    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lr4\" (UniqueName: \"kubernetes.io/projected/96ad00cf-432f-4d56-add3-7330ca4bfa10-kube-api-access-p9lr4\") pod \"maas-api-revoke-sub-db66d052-cxrpn\" (UID: \"96ad00cf-432f-4d56-add3-7330ca4bfa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:48.609926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.603252    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db66d052-cxrpn\" (UID: \"96ad00cf-432f-4d56-add3-7330ca4bfa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:48.615768 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.615734    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db66d052-cxrpn\" (UID: \"96ad00cf-432f-4d56-add3-7330ca4bfa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:48.619737 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.619709    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lr4\" (UniqueName: \"kubernetes.io/projected/96ad00cf-432f-4d56-add3-7330ca4bfa10-kube-api-access-p9lr4\") pod \"maas-api-revoke-sub-db66d052-cxrpn\" (UID: \"96ad00cf-432f-4d56-add3-7330ca4bfa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:48.714122 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.714035    4898 patch_prober.go:29] interesting pod/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 07:42:48.714122 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.714107    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 07:42:48.742999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:48.742974    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:49.087876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:49.087846    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"]
Nov 02 07:42:49.090584 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:42:49.090551    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ad00cf_432f_4d56_add3_7330ca4bfa10.slice/crio-4336f7c27e612856d823ea5c14c3768b92dd62952628b53a89bf530637272fef WatchSource:0}: Error finding container 4336f7c27e612856d823ea5c14c3768b92dd62952628b53a89bf530637272fef: Status 404 returned error can't find the container with id 4336f7c27e612856d823ea5c14c3768b92dd62952628b53a89bf530637272fef
Nov 02 07:42:49.143069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:49.143039    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn" event={"ID":"96ad00cf-432f-4d56-add3-7330ca4bfa10","Type":"ContainerStarted","Data":"4336f7c27e612856d823ea5c14c3768b92dd62952628b53a89bf530637272fef"}
Nov 02 07:42:49.345254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:49.345178    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:49.713858 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:49.713817    4898 patch_prober.go:29] interesting pod/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 07:42:49.714365 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:49.713872    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 07:42:50.152874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:50.152834    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn" event={"ID":"96ad00cf-432f-4d56-add3-7330ca4bfa10","Type":"ContainerStarted","Data":"288ba54fa86b36b29979d999d38551f242ea4a5a8eb52b044cb1529d2377c925"}
Nov 02 07:42:50.200028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:50.199965    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn" podStartSLOduration=2.199946234 podStartE2EDuration="2.199946234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:42:50.198410557 +0000 UTC m=+1120.888737541" watchObservedRunningTime="2026-11-02 07:42:50.199946234 +0000 UTC m=+1120.890273197"
Nov 02 07:42:50.719141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:50.719100    4898 patch_prober.go:29] interesting pod/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 07:42:50.720323 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:50.720281    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 07:42:51.194008 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:51.193968    4898 generic.go:356] "Generic (PLEG): container finished" podID="96ad00cf-432f-4d56-add3-7330ca4bfa10" containerID="288ba54fa86b36b29979d999d38551f242ea4a5a8eb52b044cb1529d2377c925" exitCode=0
Nov 02 07:42:51.194311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:51.194285    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn" event={"ID":"96ad00cf-432f-4d56-add3-7330ca4bfa10","Type":"ContainerDied","Data":"288ba54fa86b36b29979d999d38551f242ea4a5a8eb52b044cb1529d2377c925"}
Nov 02 07:42:51.714284 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:51.714183    4898 patch_prober.go:29] interesting pod/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 07:42:51.714449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:51.714275    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 07:42:52.437257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:52.437225    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:52.498798 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:52.498721    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-openshift-ca\") pod \"96ad00cf-432f-4d56-add3-7330ca4bfa10\" (UID: \"96ad00cf-432f-4d56-add3-7330ca4bfa10\") "
Nov 02 07:42:52.498996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:52.498805    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-xks-ca\") pod \"96ad00cf-432f-4d56-add3-7330ca4bfa10\" (UID: \"96ad00cf-432f-4d56-add3-7330ca4bfa10\") "
Nov 02 07:42:52.498996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:52.498841    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ad00cf-432f-4d56-add3-7330ca4bfa10-kube-api-access-p9lr4\" (UniqueName: \"kubernetes.io/projected/96ad00cf-432f-4d56-add3-7330ca4bfa10-kube-api-access-p9lr4\") pod \"96ad00cf-432f-4d56-add3-7330ca4bfa10\" (UID: \"96ad00cf-432f-4d56-add3-7330ca4bfa10\") "
Nov 02 07:42:52.501267 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:52.501119    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-openshift-ca" pod "96ad00cf-432f-4d56-add3-7330ca4bfa10" (UID: "96ad00cf-432f-4d56-add3-7330ca4bfa10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:42:52.502597 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:52.502571    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ad00cf-432f-4d56-add3-7330ca4bfa10-kube-api-access-p9lr4" pod "96ad00cf-432f-4d56-add3-7330ca4bfa10" (UID: "96ad00cf-432f-4d56-add3-7330ca4bfa10"). InnerVolumeSpecName "kube-api-access-p9lr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:42:52.502941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:52.502874    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-xks-ca" pod "96ad00cf-432f-4d56-add3-7330ca4bfa10" (UID: "96ad00cf-432f-4d56-add3-7330ca4bfa10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:42:52.602912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:52.602453    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:42:52.602912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:52.602494    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ad00cf-432f-4d56-add3-7330ca4bfa10-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:42:52.602912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:52.602508    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9lr4\" (UniqueName: \"kubernetes.io/projected/96ad00cf-432f-4d56-add3-7330ca4bfa10-kube-api-access-p9lr4\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:42:52.718577 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:52.718546    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:52.719688 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:52.719664    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:42:53.216224 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:53.216165    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"
Nov 02 07:42:53.216400 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:53.216164    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn" event={"ID":"96ad00cf-432f-4d56-add3-7330ca4bfa10","Type":"ContainerDied","Data":"4336f7c27e612856d823ea5c14c3768b92dd62952628b53a89bf530637272fef"}
Nov 02 07:42:53.216400 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:53.216346    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4336f7c27e612856d823ea5c14c3768b92dd62952628b53a89bf530637272fef"
Nov 02 07:42:54.033878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:54.033841    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"]
Nov 02 07:42:54.044718 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:54.044688    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17be9830-fcwsh"]
Nov 02 07:42:55.389348 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.389316    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"]
Nov 02 07:42:55.390050 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.390033    4898 cpu_manager.go:414] "containerMap: removing container" podUID="96ad00cf-432f-4d56-add3-7330ca4bfa10" containerName="revoke-keys"
Nov 02 07:42:55.390050 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.390051    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ad00cf-432f-4d56-add3-7330ca4bfa10" containerName="revoke-keys"
Nov 02 07:42:55.390189 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.390179    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ad00cf-432f-4d56-add3-7330ca4bfa10" containerName="revoke-keys"
Nov 02 07:42:55.394343 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.394322    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:42:55.415773 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.415745    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"]
Nov 02 07:42:55.421361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.421339    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:42:55.421531 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.421490    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:42:55.421646 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.421575    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:42:55.423016 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.422998    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:42:55.484711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.484671    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f5d62b-8341-483f-8f30-1c96b01001c8" path="/var/lib/kubelet/pods/b2f5d62b-8341-483f-8f30-1c96b01001c8/volumes"
Nov 02 07:42:55.547445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.547413    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlr7c\" (UniqueName: \"kubernetes.io/projected/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-kube-api-access-dlr7c\") pod \"maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m\" (UID: \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:42:55.547676 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.547649    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m\" (UID: \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:42:55.547788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.547762    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m\" (UID: \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:42:55.649155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.649059    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m\" (UID: \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:42:55.649155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.649123    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m\" (UID: \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:42:55.649394 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.649281    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlr7c\" (UniqueName: \"kubernetes.io/projected/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-kube-api-access-dlr7c\") pod \"maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m\" (UID: \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:42:55.649855 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.649826    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m\" (UID: \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:42:55.653413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.653385    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m\" (UID: \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:42:55.673161 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.673129    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlr7c\" (UniqueName: \"kubernetes.io/projected/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-kube-api-access-dlr7c\") pod \"maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m\" (UID: \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:42:55.711598 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:55.711570    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:42:56.029318 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:42:56.029283    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8128d4a_fb14_4dd1_ba28_8c6f258f302a.slice/crio-088a9c1f977b9035372a339ecf453d88e4335468914ef1920db2a4ec9f80ec2e WatchSource:0}: Error finding container 088a9c1f977b9035372a339ecf453d88e4335468914ef1920db2a4ec9f80ec2e: Status 404 returned error can't find the container with id 088a9c1f977b9035372a339ecf453d88e4335468914ef1920db2a4ec9f80ec2e
Nov 02 07:42:56.034522 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:56.034011    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"]
Nov 02 07:42:56.245097 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:56.245059    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m" event={"ID":"d8128d4a-fb14-4dd1-ba28-8c6f258f302a","Type":"ContainerStarted","Data":"088a9c1f977b9035372a339ecf453d88e4335468914ef1920db2a4ec9f80ec2e"}
Nov 02 07:42:57.255253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:57.255177    4898 generic.go:356] "Generic (PLEG): container finished" podID="d8128d4a-fb14-4dd1-ba28-8c6f258f302a" containerID="64becaaddcdc711dbad1b09e147c50b17bcca28611a024ad9e625e38c48dfe3b" exitCode=0
Nov 02 07:42:57.255629 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:57.255266    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m" event={"ID":"d8128d4a-fb14-4dd1-ba28-8c6f258f302a","Type":"ContainerDied","Data":"64becaaddcdc711dbad1b09e147c50b17bcca28611a024ad9e625e38c48dfe3b"}
Nov 02 07:42:58.346757 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:58.346723    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"]
Nov 02 07:42:58.394781 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:58.385631    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"]
Nov 02 07:42:58.472252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:58.472226    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:42:58.600959 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:58.600346    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-openshift-ca\") pod \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\" (UID: \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\") "
Nov 02 07:42:58.600959 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:58.600940    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-openshift-ca" pod "d8128d4a-fb14-4dd1-ba28-8c6f258f302a" (UID: "d8128d4a-fb14-4dd1-ba28-8c6f258f302a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:42:58.608912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:58.601764    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-kube-api-access-dlr7c\" (UniqueName: \"kubernetes.io/projected/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-kube-api-access-dlr7c\") pod \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\" (UID: \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\") "
Nov 02 07:42:58.608912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:58.602136    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-xks-ca\") pod \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\" (UID: \"d8128d4a-fb14-4dd1-ba28-8c6f258f302a\") "
Nov 02 07:42:58.608912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:58.606965    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:42:58.611955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:58.611856    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-xks-ca" pod "d8128d4a-fb14-4dd1-ba28-8c6f258f302a" (UID: "d8128d4a-fb14-4dd1-ba28-8c6f258f302a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:42:58.614503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:58.614471    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-kube-api-access-dlr7c" pod "d8128d4a-fb14-4dd1-ba28-8c6f258f302a" (UID: "d8128d4a-fb14-4dd1-ba28-8c6f258f302a"). InnerVolumeSpecName "kube-api-access-dlr7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:42:58.708236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:58.708203    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:42:58.708236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:58.708231    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlr7c\" (UniqueName: \"kubernetes.io/projected/d8128d4a-fb14-4dd1-ba28-8c6f258f302a-kube-api-access-dlr7c\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:42:59.278955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:59.278915    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m" event={"ID":"d8128d4a-fb14-4dd1-ba28-8c6f258f302a","Type":"ContainerDied","Data":"088a9c1f977b9035372a339ecf453d88e4335468914ef1920db2a4ec9f80ec2e"}
Nov 02 07:42:59.279160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:59.278972    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="088a9c1f977b9035372a339ecf453d88e4335468914ef1920db2a4ec9f80ec2e"
Nov 02 07:42:59.279160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:42:59.279042    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"
Nov 02 07:43:03.389197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:03.389060    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m" podUID="0a40ebf4-6e54-498e-b5bd-837dcea6b6f5" containerName="payload-processing" containerID="cri-o://c30a18352b08055329425a821880d0b05003f6888c3e4ddc32daf1c97e265a32" gracePeriod=25
Nov 02 07:43:03.415016 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:03.410063    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8" podUID="83278155-b02a-4944-ae0b-6d3c215c33a7" containerName="payload-pre-processing" containerID="cri-o://ac84b145294d35fb64bde47b319597c6a32a3a81adc51cf3f30668beb5700640" gracePeriod=25
Nov 02 07:43:03.436913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:03.417250    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:43:03.543107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:03.523611    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:43:03.568918 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:03.554068    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:43:03.879028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:03.878975    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:43:03.940232 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:03.940109    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 02 07:43:03.963008 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:03.962963    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:43:04.032599 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.031823    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:43:04.095910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.094974    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 07:43:04.435992 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.435813    4898 generic.go:356] "Generic (PLEG): container finished" podID="83278155-b02a-4944-ae0b-6d3c215c33a7" containerID="ac84b145294d35fb64bde47b319597c6a32a3a81adc51cf3f30668beb5700640" exitCode=0
Nov 02 07:43:04.435992 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.435921    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8" event={"ID":"83278155-b02a-4944-ae0b-6d3c215c33a7","Type":"ContainerDied","Data":"ac84b145294d35fb64bde47b319597c6a32a3a81adc51cf3f30668beb5700640"}
Nov 02 07:43:04.451825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.451144    4898 generic.go:356] "Generic (PLEG): container finished" podID="0a40ebf4-6e54-498e-b5bd-837dcea6b6f5" containerID="c30a18352b08055329425a821880d0b05003f6888c3e4ddc32daf1c97e265a32" exitCode=0
Nov 02 07:43:04.451825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.451259    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m" event={"ID":"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5","Type":"ContainerDied","Data":"c30a18352b08055329425a821880d0b05003f6888c3e4ddc32daf1c97e265a32"}
Nov 02 07:43:04.534816 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.534780    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 02 07:43:04.535559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.535538    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:43:04.655293 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.649234    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:43:04.740932 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.736697    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"
Nov 02 07:43:04.916290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.904048    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83278155-b02a-4944-ae0b-6d3c215c33a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83278155-b02a-4944-ae0b-6d3c215c33a7-plugins-config-volume\") pod \"83278155-b02a-4944-ae0b-6d3c215c33a7\" (UID: \"83278155-b02a-4944-ae0b-6d3c215c33a7\") "
Nov 02 07:43:04.916290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.904382    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83278155-b02a-4944-ae0b-6d3c215c33a7-kube-api-access-mt6zw\" (UniqueName: \"kubernetes.io/projected/83278155-b02a-4944-ae0b-6d3c215c33a7-kube-api-access-mt6zw\") pod \"83278155-b02a-4944-ae0b-6d3c215c33a7\" (UID: \"83278155-b02a-4944-ae0b-6d3c215c33a7\") "
Nov 02 07:43:04.921836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.921395    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83278155-b02a-4944-ae0b-6d3c215c33a7-plugins-config-volume" pod "83278155-b02a-4944-ae0b-6d3c215c33a7" (UID: "83278155-b02a-4944-ae0b-6d3c215c33a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:04.939003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.938726    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83278155-b02a-4944-ae0b-6d3c215c33a7-kube-api-access-mt6zw" pod "83278155-b02a-4944-ae0b-6d3c215c33a7" (UID: "83278155-b02a-4944-ae0b-6d3c215c33a7"). InnerVolumeSpecName "kube-api-access-mt6zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:04.954298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:04.954275    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"
Nov 02 07:43:05.007248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.007213    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt6zw\" (UniqueName: \"kubernetes.io/projected/83278155-b02a-4944-ae0b-6d3c215c33a7-kube-api-access-mt6zw\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:05.007248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.007250    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83278155-b02a-4944-ae0b-6d3c215c33a7-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:05.108054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.108024    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-plugins-config-volume\") pod \"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5\" (UID: \"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5\") "
Nov 02 07:43:05.108268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.108100    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-kube-api-access-cffjn\" (UniqueName: \"kubernetes.io/projected/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-kube-api-access-cffjn\") pod \"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5\" (UID: \"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5\") "
Nov 02 07:43:05.108436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.108403    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-plugins-config-volume" pod "0a40ebf4-6e54-498e-b5bd-837dcea6b6f5" (UID: "0a40ebf4-6e54-498e-b5bd-837dcea6b6f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:05.108641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.108625    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:05.110318 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.110298    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-kube-api-access-cffjn" pod "0a40ebf4-6e54-498e-b5bd-837dcea6b6f5" (UID: "0a40ebf4-6e54-498e-b5bd-837dcea6b6f5"). InnerVolumeSpecName "kube-api-access-cffjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:05.210488 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.210409    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cffjn\" (UniqueName: \"kubernetes.io/projected/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5-kube-api-access-cffjn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:05.461943 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.461848    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"
Nov 02 07:43:05.473680 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.471826    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"
Nov 02 07:43:05.475249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.473878    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8" event={"ID":"83278155-b02a-4944-ae0b-6d3c215c33a7","Type":"ContainerDied","Data":"247b8ae1e44f632e6da0bd462e5efd52b01e18108977ebe0fb7fe76e912db0bc"}
Nov 02 07:43:05.475249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.474055    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m" event={"ID":"0a40ebf4-6e54-498e-b5bd-837dcea6b6f5","Type":"ContainerDied","Data":"14e757ae363dee0a97a68d2e820b1d45c948459a3606ce1b043e7ad36fa89dda"}
Nov 02 07:43:05.475249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.474095    4898 scope.go:122] "RemoveContainer" containerID="ac84b145294d35fb64bde47b319597c6a32a3a81adc51cf3f30668beb5700640"
Nov 02 07:43:05.496744 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.496721    4898 scope.go:122] "RemoveContainer" containerID="c30a18352b08055329425a821880d0b05003f6888c3e4ddc32daf1c97e265a32"
Nov 02 07:43:05.541682 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.541655    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"]
Nov 02 07:43:05.553433 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.553409    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fe085e-7fd6c7c44c-k75k8"]
Nov 02 07:43:05.582053 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.582026    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"]
Nov 02 07:43:05.591705 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:05.591679    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe085e-6fd59cf677-zlp7m"]
Nov 02 07:43:07.277585 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.277545    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"]
Nov 02 07:43:07.278565 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.278537    4898 cpu_manager.go:414] "containerMap: removing container" podUID="83278155-b02a-4944-ae0b-6d3c215c33a7" containerName="payload-pre-processing"
Nov 02 07:43:07.278565 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.278562    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83278155-b02a-4944-ae0b-6d3c215c33a7" containerName="payload-pre-processing"
Nov 02 07:43:07.278874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.278589    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d8128d4a-fb14-4dd1-ba28-8c6f258f302a" containerName="revoke-keys"
Nov 02 07:43:07.278874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.278597    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8128d4a-fb14-4dd1-ba28-8c6f258f302a" containerName="revoke-keys"
Nov 02 07:43:07.278874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.278654    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0a40ebf4-6e54-498e-b5bd-837dcea6b6f5" containerName="payload-processing"
Nov 02 07:43:07.278874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.278663    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a40ebf4-6e54-498e-b5bd-837dcea6b6f5" containerName="payload-processing"
Nov 02 07:43:07.278874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.278821    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a40ebf4-6e54-498e-b5bd-837dcea6b6f5" containerName="payload-processing"
Nov 02 07:43:07.278874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.278838    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="83278155-b02a-4944-ae0b-6d3c215c33a7" containerName="payload-pre-processing"
Nov 02 07:43:07.278874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.278852    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8128d4a-fb14-4dd1-ba28-8c6f258f302a" containerName="revoke-keys"
Nov 02 07:43:07.295957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.285242    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.320939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.320607    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"]
Nov 02 07:43:07.344293 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.344259    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.347118 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347087    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"]
Nov 02 07:43:07.347327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347124    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.347327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347241    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.347327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347317    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.347756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347348    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.347756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347376    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjfgp\" (UniqueName: \"kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-kube-api-access-cjfgp\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.347756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347404    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4df2b59-c602-426a-b7f9-debe6dfce67c-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.347756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347447    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5swcn\" (UniqueName: \"kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-kube-api-access-5swcn\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.347756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347477    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.347756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347517    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.347756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347562    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.347756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347588    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.347756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347634    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.347756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347666    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.347756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347751    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.348407 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.347828    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.348407 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.348108    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.348407 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.348242    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.348407 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.348287    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.373677 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.372590    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"]
Nov 02 07:43:07.449482 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449452    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.449634 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449489    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.449634 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449517    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.449764 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449638    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.449764 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449706    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.449764 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449731    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.449764 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449748    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjfgp\" (UniqueName: \"kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-kube-api-access-cjfgp\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.449998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449765    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4df2b59-c602-426a-b7f9-debe6dfce67c-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.449998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449797    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5swcn\" (UniqueName: \"kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-kube-api-access-5swcn\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.449998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449817    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.449998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449843    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.449998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449885    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.449998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449904    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.449998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449953    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.449998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.449981    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.450357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.450007    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.450357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.450039    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.450357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.450131    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.450357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.450174    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.450357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.450214    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.450357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.450347    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.450698 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.450679    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4df2b59-c602-426a-b7f9-debe6dfce67c-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.450827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.450805    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.451131 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.451105    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.451248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.451117    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.451248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.451197    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.451248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.451220    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.451401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.451269    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.452627 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.452489    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.452627 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.452510    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.453316 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.453295    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.453380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.453360    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.470067 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.470039    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a40ebf4-6e54-498e-b5bd-837dcea6b6f5" path="/var/lib/kubelet/pods/0a40ebf4-6e54-498e-b5bd-837dcea6b6f5/volumes"
Nov 02 07:43:07.470662 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.470641    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83278155-b02a-4944-ae0b-6d3c215c33a7" path="/var/lib/kubelet/pods/83278155-b02a-4944-ae0b-6d3c215c33a7/volumes"
Nov 02 07:43:07.489077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.488997    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.489180 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.489107    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5swcn\" (UniqueName: \"kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-kube-api-access-5swcn\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.489180 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.489122    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjfgp\" (UniqueName: \"kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-kube-api-access-cjfgp\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.491441 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.491412    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.616445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.616366    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:07.676452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.676398    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:07.943533 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:43:07.943503    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37f72a4_f404_47fd_9c2a_d5485fe59a49.slice/crio-62bba73e32328f887ba39d8bb57d380a6644eed8751ed2e5d06afde0fbc5da21 WatchSource:0}: Error finding container 62bba73e32328f887ba39d8bb57d380a6644eed8751ed2e5d06afde0fbc5da21: Status 404 returned error can't find the container with id 62bba73e32328f887ba39d8bb57d380a6644eed8751ed2e5d06afde0fbc5da21
Nov 02 07:43:07.951428 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.951398    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"]
Nov 02 07:43:07.952653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.952603    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:07.952778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.952749    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:07.953006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:07.952975    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:08.024017 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:43:08.023989    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4df2b59_c602_426a_b7f9_debe6dfce67c.slice/crio-e447ae09a5e1abd46e5e405201997c26f8023262ca822eee68e7bc7717fb5994 WatchSource:0}: Error finding container e447ae09a5e1abd46e5e405201997c26f8023262ca822eee68e7bc7717fb5994: Status 404 returned error can't find the container with id e447ae09a5e1abd46e5e405201997c26f8023262ca822eee68e7bc7717fb5994
Nov 02 07:43:08.026974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:08.026932    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:08.027051 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:08.027025    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:08.027117 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:08.027069    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:43:08.027937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:08.027915    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"]
Nov 02 07:43:08.510723 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:08.510680    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5" event={"ID":"a37f72a4-f404-47fd-9c2a-d5485fe59a49","Type":"ContainerStarted","Data":"936ebd104b41cc058dd75b209e9dff169f981a5dc11ac6cc40ea10f1fc7601bf"}
Nov 02 07:43:08.511226 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:08.510748    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5" event={"ID":"a37f72a4-f404-47fd-9c2a-d5485fe59a49","Type":"ContainerStarted","Data":"62bba73e32328f887ba39d8bb57d380a6644eed8751ed2e5d06afde0fbc5da21"}
Nov 02 07:43:08.512657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:08.512630    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" event={"ID":"c4df2b59-c602-426a-b7f9-debe6dfce67c","Type":"ContainerStarted","Data":"f426e7fb0c5bab528775e3cc974786c2d687ef9e6af751ab6486cbc51aa71f98"}
Nov 02 07:43:08.512795 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:08.512674    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" event={"ID":"c4df2b59-c602-426a-b7f9-debe6dfce67c","Type":"ContainerStarted","Data":"e447ae09a5e1abd46e5e405201997c26f8023262ca822eee68e7bc7717fb5994"}
Nov 02 07:43:08.616457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:08.616423    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:08.635261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:08.635198    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" podStartSLOduration=1.635178368 podStartE2EDuration="1.635178368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:08.629070624 +0000 UTC m=+1139.319397589" watchObservedRunningTime="2026-11-02 07:43:08.635178368 +0000 UTC m=+1139.325505331"
Nov 02 07:43:08.637449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:08.637372    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5" podStartSLOduration=1.6373558529999999 podStartE2EDuration="1.637355853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:43:08.581870531 +0000 UTC m=+1139.272197493" watchObservedRunningTime="2026-11-02 07:43:08.637355853 +0000 UTC m=+1139.327682817"
Nov 02 07:43:08.679429 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:08.679308    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:09.622342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:09.622298    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 07:43:09.622829 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:09.622364    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5" podUID="a37f72a4-f404-47fd-9c2a-d5485fe59a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 07:43:09.683941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:09.682010    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:43:09.683941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:09.682072    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" podUID="c4df2b59-c602-426a-b7f9-debe6dfce67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 02 07:43:09.746749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:09.746716    4898 scope.go:122] "RemoveContainer" containerID="c47e40b8d2523bcd35eb68524f434f228f79f63d66959def2c4548d1f7f1978b"
Nov 02 07:43:09.787102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:09.785736    4898 scope.go:122] "RemoveContainer" containerID="4c7e2ad0a9b80c08b98daa90b00a3b63ccc5726aaaf0a71976f403ea574c2ef6"
Nov 02 07:43:10.642910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:10.634471    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 07:43:10.642910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:10.634571    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5" podUID="a37f72a4-f404-47fd-9c2a-d5485fe59a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 07:43:10.686912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:10.682326    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:43:10.686912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:10.682432    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" podUID="c4df2b59-c602-426a-b7f9-debe6dfce67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 02 07:43:11.617604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:11.617568    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 07:43:11.617852 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:11.617628    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5" podUID="a37f72a4-f404-47fd-9c2a-d5485fe59a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 07:43:11.685373 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:11.685337    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:43:11.686014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:11.685972    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" podUID="c4df2b59-c602-426a-b7f9-debe6dfce67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 02 07:43:12.617646 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:12.617612    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 07:43:12.617833 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:12.617666    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5" podUID="a37f72a4-f404-47fd-9c2a-d5485fe59a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 07:43:12.684043 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:12.684005    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:43:12.684422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:12.684386    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" podUID="c4df2b59-c602-426a-b7f9-debe6dfce67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 02 07:43:13.630847 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:13.630810    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 07:43:13.631617 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:13.631079    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5" podUID="a37f72a4-f404-47fd-9c2a-d5485fe59a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 07:43:13.708203 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:13.708136    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:43:13.708501 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:13.708457    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" podUID="c4df2b59-c602-426a-b7f9-debe6dfce67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 02 07:43:14.617420 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:14.617371    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 02 07:43:14.617600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:14.617447    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5" podUID="a37f72a4-f404-47fd-9c2a-d5485fe59a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 02 07:43:14.679298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:14.679259    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:43:14.679777 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:14.679324    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" podUID="c4df2b59-c602-426a-b7f9-debe6dfce67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 02 07:43:14.994180 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:43:14.994140    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:43:15.679205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:15.679158    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 02 07:43:15.679486 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:15.679455    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" podUID="c4df2b59-c602-426a-b7f9-debe6dfce67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 02 07:43:16.145550 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.145474    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:16.147181 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.147161    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:16.147785 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.147761    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:43:16.412637 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.412601    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"]
Nov 02 07:43:16.419960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.419935    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:16.429812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.429777    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f8ece77-nlmzh\" (UID: \"b4e01286-cce2-4704-b024-cb73980f5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:16.429976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.429910    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxn2\" (UniqueName: \"kubernetes.io/projected/b4e01286-cce2-4704-b024-cb73980f5d9a-kube-api-access-prxn2\") pod \"maas-api-revoke-sub-1f8ece77-nlmzh\" (UID: \"b4e01286-cce2-4704-b024-cb73980f5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:16.430195 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.430173    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f8ece77-nlmzh\" (UID: \"b4e01286-cce2-4704-b024-cb73980f5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:16.438546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.438525    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:43:16.443301 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.443275    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:43:16.443408 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.443386    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:43:16.443473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.443432    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:43:16.453724 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.453700    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"]
Nov 02 07:43:16.531730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.531695    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f8ece77-nlmzh\" (UID: \"b4e01286-cce2-4704-b024-cb73980f5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:16.531953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.531790    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxn2\" (UniqueName: \"kubernetes.io/projected/b4e01286-cce2-4704-b024-cb73980f5d9a-kube-api-access-prxn2\") pod \"maas-api-revoke-sub-1f8ece77-nlmzh\" (UID: \"b4e01286-cce2-4704-b024-cb73980f5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:16.532228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.532198    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f8ece77-nlmzh\" (UID: \"b4e01286-cce2-4704-b024-cb73980f5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:16.532954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.532931    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f8ece77-nlmzh\" (UID: \"b4e01286-cce2-4704-b024-cb73980f5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:16.534333 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.534312    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f8ece77-nlmzh\" (UID: \"b4e01286-cce2-4704-b024-cb73980f5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:16.563026 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.563003    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxn2\" (UniqueName: \"kubernetes.io/projected/b4e01286-cce2-4704-b024-cb73980f5d9a-kube-api-access-prxn2\") pod \"maas-api-revoke-sub-1f8ece77-nlmzh\" (UID: \"b4e01286-cce2-4704-b024-cb73980f5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:16.682099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.682018    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:16.682571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.682551    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:16.683402 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.683378    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:43:16.747443 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:16.747420    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:17.081782 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:43:17.081571    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e01286_cce2_4704_b024_cb73980f5d9a.slice/crio-80ab0c0bbe4cec835d30b70349e78e16bcbf19f6f4e1ea4ef355c8cc659ed549 WatchSource:0}: Error finding container 80ab0c0bbe4cec835d30b70349e78e16bcbf19f6f4e1ea4ef355c8cc659ed549: Status 404 returned error can't find the container with id 80ab0c0bbe4cec835d30b70349e78e16bcbf19f6f4e1ea4ef355c8cc659ed549
Nov 02 07:43:17.083853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:17.083828    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"]
Nov 02 07:43:17.123798 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:17.123769    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh" event={"ID":"b4e01286-cce2-4704-b024-cb73980f5d9a","Type":"ContainerStarted","Data":"80ab0c0bbe4cec835d30b70349e78e16bcbf19f6f4e1ea4ef355c8cc659ed549"}
Nov 02 07:43:17.960257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:17.960226    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"]
Nov 02 07:43:18.136208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:18.136154    4898 generic.go:356] "Generic (PLEG): container finished" podID="b4e01286-cce2-4704-b024-cb73980f5d9a" containerID="f578a22fdcbc0d1a63976acd42cfa9a4a12b211239c867f78658049016136bf7" exitCode=0
Nov 02 07:43:18.136350 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:18.136238    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh" event={"ID":"b4e01286-cce2-4704-b024-cb73980f5d9a","Type":"ContainerDied","Data":"f578a22fdcbc0d1a63976acd42cfa9a4a12b211239c867f78658049016136bf7"}
Nov 02 07:43:19.353627 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:19.353596    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:19.402374 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:19.402338    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-openshift-ca\") pod \"b4e01286-cce2-4704-b024-cb73980f5d9a\" (UID: \"b4e01286-cce2-4704-b024-cb73980f5d9a\") "
Nov 02 07:43:19.402553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:19.402456    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e01286-cce2-4704-b024-cb73980f5d9a-kube-api-access-prxn2\" (UniqueName: \"kubernetes.io/projected/b4e01286-cce2-4704-b024-cb73980f5d9a-kube-api-access-prxn2\") pod \"b4e01286-cce2-4704-b024-cb73980f5d9a\" (UID: \"b4e01286-cce2-4704-b024-cb73980f5d9a\") "
Nov 02 07:43:19.402628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:19.402584    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-xks-ca\") pod \"b4e01286-cce2-4704-b024-cb73980f5d9a\" (UID: \"b4e01286-cce2-4704-b024-cb73980f5d9a\") "
Nov 02 07:43:19.403151 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:19.403124    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-openshift-ca" pod "b4e01286-cce2-4704-b024-cb73980f5d9a" (UID: "b4e01286-cce2-4704-b024-cb73980f5d9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:19.403365 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:19.403343    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:19.405302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:19.405270    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-xks-ca" pod "b4e01286-cce2-4704-b024-cb73980f5d9a" (UID: "b4e01286-cce2-4704-b024-cb73980f5d9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:43:19.405392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:19.405333    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e01286-cce2-4704-b024-cb73980f5d9a-kube-api-access-prxn2" pod "b4e01286-cce2-4704-b024-cb73980f5d9a" (UID: "b4e01286-cce2-4704-b024-cb73980f5d9a"). InnerVolumeSpecName "kube-api-access-prxn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:19.504979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:19.504947    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e01286-cce2-4704-b024-cb73980f5d9a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:19.504979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:19.504981    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prxn2\" (UniqueName: \"kubernetes.io/projected/b4e01286-cce2-4704-b024-cb73980f5d9a-kube-api-access-prxn2\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:20.157614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:20.157583    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"
Nov 02 07:43:20.157766 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:20.157584    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh" event={"ID":"b4e01286-cce2-4704-b024-cb73980f5d9a","Type":"ContainerDied","Data":"80ab0c0bbe4cec835d30b70349e78e16bcbf19f6f4e1ea4ef355c8cc659ed549"}
Nov 02 07:43:20.157766 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:20.157720    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80ab0c0bbe4cec835d30b70349e78e16bcbf19f6f4e1ea4ef355c8cc659ed549"
Nov 02 07:43:22.965915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:22.962955    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8" podUID="d3ae1d71-59c1-4344-ad51-f0e815a60c9c" containerName="payload-pre-processing" containerID="cri-o://1a8fc3790b1048734e68e0bef35b6bfbf086b4ac6b77630a4618f0c1270af624" gracePeriod=25
Nov 02 07:43:23.188053 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:23.188018    4898 generic.go:356] "Generic (PLEG): container finished" podID="d3ae1d71-59c1-4344-ad51-f0e815a60c9c" containerID="1a8fc3790b1048734e68e0bef35b6bfbf086b4ac6b77630a4618f0c1270af624" exitCode=0
Nov 02 07:43:23.188242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:23.188084    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8" event={"ID":"d3ae1d71-59c1-4344-ad51-f0e815a60c9c","Type":"ContainerDied","Data":"1a8fc3790b1048734e68e0bef35b6bfbf086b4ac6b77630a4618f0c1270af624"}
Nov 02 07:43:23.319695 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:23.319665    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"
Nov 02 07:43:23.467678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:23.465239    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-plugins-config-volume\") pod \"d3ae1d71-59c1-4344-ad51-f0e815a60c9c\" (UID: \"d3ae1d71-59c1-4344-ad51-f0e815a60c9c\") "
Nov 02 07:43:23.467678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:23.465314    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-kube-api-access-z4tv5\" (UniqueName: \"kubernetes.io/projected/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-kube-api-access-z4tv5\") pod \"d3ae1d71-59c1-4344-ad51-f0e815a60c9c\" (UID: \"d3ae1d71-59c1-4344-ad51-f0e815a60c9c\") "
Nov 02 07:43:23.467678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:23.466056    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-plugins-config-volume" pod "d3ae1d71-59c1-4344-ad51-f0e815a60c9c" (UID: "d3ae1d71-59c1-4344-ad51-f0e815a60c9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:23.467678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:23.466168    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:23.469622 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:23.469577    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-kube-api-access-z4tv5" pod "d3ae1d71-59c1-4344-ad51-f0e815a60c9c" (UID: "d3ae1d71-59c1-4344-ad51-f0e815a60c9c"). InnerVolumeSpecName "kube-api-access-z4tv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:23.568137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:23.568057    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4tv5\" (UniqueName: \"kubernetes.io/projected/d3ae1d71-59c1-4344-ad51-f0e815a60c9c-kube-api-access-z4tv5\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:24.197554 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:24.197521    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"
Nov 02 07:43:24.197554 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:24.197546    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8" event={"ID":"d3ae1d71-59c1-4344-ad51-f0e815a60c9c","Type":"ContainerDied","Data":"b2c5a2a2ce600f7ecd1360d64c4f343504d8c7c2625681d674b29aa24b52f9eb"}
Nov 02 07:43:24.198116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:24.197604    4898 scope.go:122] "RemoveContainer" containerID="1a8fc3790b1048734e68e0bef35b6bfbf086b4ac6b77630a4618f0c1270af624"
Nov 02 07:43:24.261114 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:24.261075    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"]
Nov 02 07:43:24.270752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:24.270727    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc2bcd-756c994b87-hv2m8"]
Nov 02 07:43:25.467544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:25.467501    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ae1d71-59c1-4344-ad51-f0e815a60c9c" path="/var/lib/kubelet/pods/d3ae1d71-59c1-4344-ad51-f0e815a60c9c/volumes"
Nov 02 07:43:33.170943 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:33.170903    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"]
Nov 02 07:43:33.222162 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:33.222122    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"]
Nov 02 07:43:33.234431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:33.234397    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77dc65fb-9jw8m"]
Nov 02 07:43:33.500014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:33.499457    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8128d4a-fb14-4dd1-ba28-8c6f258f302a" path="/var/lib/kubelet/pods/d8128d4a-fb14-4dd1-ba28-8c6f258f302a/volumes"
Nov 02 07:43:38.171832 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:38.171781    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm" podUID="2383994e-608b-4b83-b8fa-2a24992bc937" containerName="payload-pre-processing" containerID="cri-o://3fe5450c90beaa2f56ca79b9ecd93570cc8cd0034226197e17472ea31ffa2034" gracePeriod=25
Nov 02 07:43:38.380909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:38.376204    4898 generic.go:356] "Generic (PLEG): container finished" podID="2383994e-608b-4b83-b8fa-2a24992bc937" containerID="3fe5450c90beaa2f56ca79b9ecd93570cc8cd0034226197e17472ea31ffa2034" exitCode=0
Nov 02 07:43:38.380909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:38.376473    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm" event={"ID":"2383994e-608b-4b83-b8fa-2a24992bc937","Type":"ContainerDied","Data":"3fe5450c90beaa2f56ca79b9ecd93570cc8cd0034226197e17472ea31ffa2034"}
Nov 02 07:43:38.580162 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:38.580121    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"
Nov 02 07:43:38.675006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:38.674972    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2383994e-608b-4b83-b8fa-2a24992bc937-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2383994e-608b-4b83-b8fa-2a24992bc937-plugins-config-volume\") pod \"2383994e-608b-4b83-b8fa-2a24992bc937\" (UID: \"2383994e-608b-4b83-b8fa-2a24992bc937\") "
Nov 02 07:43:38.675248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:38.675228    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2383994e-608b-4b83-b8fa-2a24992bc937-kube-api-access-rz6gq\" (UniqueName: \"kubernetes.io/projected/2383994e-608b-4b83-b8fa-2a24992bc937-kube-api-access-rz6gq\") pod \"2383994e-608b-4b83-b8fa-2a24992bc937\" (UID: \"2383994e-608b-4b83-b8fa-2a24992bc937\") "
Nov 02 07:43:38.675528 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:38.675498    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2383994e-608b-4b83-b8fa-2a24992bc937-plugins-config-volume" pod "2383994e-608b-4b83-b8fa-2a24992bc937" (UID: "2383994e-608b-4b83-b8fa-2a24992bc937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:38.676247 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:38.676222    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2383994e-608b-4b83-b8fa-2a24992bc937-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:38.686759 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:38.686732    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2383994e-608b-4b83-b8fa-2a24992bc937-kube-api-access-rz6gq" pod "2383994e-608b-4b83-b8fa-2a24992bc937" (UID: "2383994e-608b-4b83-b8fa-2a24992bc937"). InnerVolumeSpecName "kube-api-access-rz6gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:38.777587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:38.777552    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz6gq\" (UniqueName: \"kubernetes.io/projected/2383994e-608b-4b83-b8fa-2a24992bc937-kube-api-access-rz6gq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:39.387026 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:39.386993    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm" event={"ID":"2383994e-608b-4b83-b8fa-2a24992bc937","Type":"ContainerDied","Data":"0c303de889439abae1f0d4c73f509ceeac755c41ce82abdafef0acb29255e5c0"}
Nov 02 07:43:39.387026 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:39.387025    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"
Nov 02 07:43:39.387866 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:39.387053    4898 scope.go:122] "RemoveContainer" containerID="3fe5450c90beaa2f56ca79b9ecd93570cc8cd0034226197e17472ea31ffa2034"
Nov 02 07:43:39.450207 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:39.450182    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"]
Nov 02 07:43:39.466361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:39.466337    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3e5f7-6ddfdf758c-79tkm"]
Nov 02 07:43:41.466074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:41.466033    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2383994e-608b-4b83-b8fa-2a24992bc937" path="/var/lib/kubelet/pods/2383994e-608b-4b83-b8fa-2a24992bc937/volumes"
Nov 02 07:43:44.989533 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:43:44.989496    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:43:52.172765 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:52.172730    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"]
Nov 02 07:43:52.173600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:52.173540    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy" containerID="cri-o://3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9" gracePeriod=30
Nov 02 07:43:52.721852 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:52.721802    4898 patch_prober.go:29] interesting pod/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:43:52.722036 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:52.721925    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:43:57.475264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.475233    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:43:57.543829 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.543793    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-envoy\") pod \"82360126-e2f7-4bf0-9abf-3c82168fb797\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") "
Nov 02 07:43:57.544122 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.544097    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-socket\") pod \"82360126-e2f7-4bf0-9abf-3c82168fb797\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") "
Nov 02 07:43:57.544405 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.544386    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-kube-api-access-fdk44\" (UniqueName: \"kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-kube-api-access-fdk44\") pod \"82360126-e2f7-4bf0-9abf-3c82168fb797\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") "
Nov 02 07:43:57.544552 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.544536    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-token\" (UniqueName: \"kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-token\") pod \"82360126-e2f7-4bf0-9abf-3c82168fb797\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") "
Nov 02 07:43:57.544705 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.544687    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-podinfo\") pod \"82360126-e2f7-4bf0-9abf-3c82168fb797\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") "
Nov 02 07:43:57.544834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.544816    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-data\") pod \"82360126-e2f7-4bf0-9abf-3c82168fb797\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") "
Nov 02 07:43:57.544979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.544962    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-credential-socket\") pod \"82360126-e2f7-4bf0-9abf-3c82168fb797\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") "
Nov 02 07:43:57.545152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.545135    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-certs\") pod \"82360126-e2f7-4bf0-9abf-3c82168fb797\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") "
Nov 02 07:43:57.545401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.545383    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82360126-e2f7-4bf0-9abf-3c82168fb797-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82360126-e2f7-4bf0-9abf-3c82168fb797-istiod-ca-cert\") pod \"82360126-e2f7-4bf0-9abf-3c82168fb797\" (UID: \"82360126-e2f7-4bf0-9abf-3c82168fb797\") "
Nov 02 07:43:57.547427 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.544876    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-socket" pod "82360126-e2f7-4bf0-9abf-3c82168fb797" (UID: "82360126-e2f7-4bf0-9abf-3c82168fb797"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:57.547594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.547391    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-data" pod "82360126-e2f7-4bf0-9abf-3c82168fb797" (UID: "82360126-e2f7-4bf0-9abf-3c82168fb797"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:57.547686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.547582    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-podinfo" pod "82360126-e2f7-4bf0-9abf-3c82168fb797" (UID: "82360126-e2f7-4bf0-9abf-3c82168fb797"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:43:57.548093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.548059    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-credential-socket" pod "82360126-e2f7-4bf0-9abf-3c82168fb797" (UID: "82360126-e2f7-4bf0-9abf-3c82168fb797"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:57.548310 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.548287    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-certs" pod "82360126-e2f7-4bf0-9abf-3c82168fb797" (UID: "82360126-e2f7-4bf0-9abf-3c82168fb797"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:57.549480 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.548571    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82360126-e2f7-4bf0-9abf-3c82168fb797-istiod-ca-cert" pod "82360126-e2f7-4bf0-9abf-3c82168fb797" (UID: "82360126-e2f7-4bf0-9abf-3c82168fb797"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:43:57.551955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.551926    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-token" pod "82360126-e2f7-4bf0-9abf-3c82168fb797" (UID: "82360126-e2f7-4bf0-9abf-3c82168fb797"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:57.553418 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.553372    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-envoy" pod "82360126-e2f7-4bf0-9abf-3c82168fb797" (UID: "82360126-e2f7-4bf0-9abf-3c82168fb797"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:43:57.580136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.580107    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-kube-api-access-fdk44" pod "82360126-e2f7-4bf0-9abf-3c82168fb797" (UID: "82360126-e2f7-4bf0-9abf-3c82168fb797"). InnerVolumeSpecName "kube-api-access-fdk44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:43:57.648171 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.648139    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:57.648171 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.648169    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:57.648393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.648197    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:57.648393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.648206    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82360126-e2f7-4bf0-9abf-3c82168fb797-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:57.648393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.648214    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:57.648393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.648226    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82360126-e2f7-4bf0-9abf-3c82168fb797-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:57.648393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.648238    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdk44\" (UniqueName: \"kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-kube-api-access-fdk44\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:57.648393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.648252    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:57.648393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.648265    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82360126-e2f7-4bf0-9abf-3c82168fb797-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:43:57.649111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.649081    4898 generic.go:356] "Generic (PLEG): container finished" podID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerID="3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9" exitCode=0
Nov 02 07:43:57.649189 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.649137    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" event={"ID":"82360126-e2f7-4bf0-9abf-3c82168fb797","Type":"ContainerDied","Data":"3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9"}
Nov 02 07:43:57.649189 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.649158    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"
Nov 02 07:43:57.649311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.649215    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx" event={"ID":"82360126-e2f7-4bf0-9abf-3c82168fb797","Type":"ContainerDied","Data":"9ca4716df53190b364898171748864998fc5effb1c94e10dd73476a3a801256b"}
Nov 02 07:43:57.649311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.649247    4898 scope.go:122] "RemoveContainer" containerID="3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9"
Nov 02 07:43:57.665970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.665953    4898 scope.go:122] "RemoveContainer" containerID="eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834"
Nov 02 07:43:57.678480 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.678453    4898 scope.go:122] "RemoveContainer" containerID="3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9"
Nov 02 07:43:57.678762 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:43:57.678741    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9\": container with ID starting with 3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9 not found: ID does not exist" containerID="3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9"
Nov 02 07:43:57.678864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.678773    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9"} err="failed to get container status \"3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9\": rpc error: code = NotFound desc = could not find container \"3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9\": container with ID starting with 3f0ee16fa885f2b8d8392c4793e932a634bd4c97772d80f2b6e68d093a16f0c9 not found: ID does not exist"
Nov 02 07:43:57.678864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.678793    4898 scope.go:122] "RemoveContainer" containerID="eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834"
Nov 02 07:43:57.679060 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:43:57.679045    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834\": container with ID starting with eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834 not found: ID does not exist" containerID="eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834"
Nov 02 07:43:57.679104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.679063    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834"} err="failed to get container status \"eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834\": rpc error: code = NotFound desc = could not find container \"eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834\": container with ID starting with eacbe35309ace598579b072dc3e886ddc58e421c725d6d43e8ddd77e37e64834 not found: ID does not exist"
Nov 02 07:43:57.708560 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.708482    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"]
Nov 02 07:43:57.716549 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:57.715117    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b3e5f7-openshift-default-84d7d984cf-dmbjx"]
Nov 02 07:43:59.563554 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:43:59.546389    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" path="/var/lib/kubelet/pods/82360126-e2f7-4bf0-9abf-3c82168fb797/volumes"
Nov 02 07:44:01.010413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.010378    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"]
Nov 02 07:44:01.011164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011141    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b4e01286-cce2-4704-b024-cb73980f5d9a" containerName="revoke-keys"
Nov 02 07:44:01.011270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011165    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e01286-cce2-4704-b024-cb73980f5d9a" containerName="revoke-keys"
Nov 02 07:44:01.011270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011183    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d3ae1d71-59c1-4344-ad51-f0e815a60c9c" containerName="payload-pre-processing"
Nov 02 07:44:01.011270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011199    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ae1d71-59c1-4344-ad51-f0e815a60c9c" containerName="payload-pre-processing"
Nov 02 07:44:01.011270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011225    4898 cpu_manager.go:414] "containerMap: removing container" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy"
Nov 02 07:44:01.011270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011231    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy"
Nov 02 07:44:01.011270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011248    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2383994e-608b-4b83-b8fa-2a24992bc937" containerName="payload-pre-processing"
Nov 02 07:44:01.011270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011253    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2383994e-608b-4b83-b8fa-2a24992bc937" containerName="payload-pre-processing"
Nov 02 07:44:01.011496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011406    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy"
Nov 02 07:44:01.011496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011422    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e01286-cce2-4704-b024-cb73980f5d9a" containerName="revoke-keys"
Nov 02 07:44:01.011496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011433    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ae1d71-59c1-4344-ad51-f0e815a60c9c" containerName="payload-pre-processing"
Nov 02 07:44:01.011496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011444    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy"
Nov 02 07:44:01.011496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.011454    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2383994e-608b-4b83-b8fa-2a24992bc937" containerName="payload-pre-processing"
Nov 02 07:44:01.015306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.015287    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"
Nov 02 07:44:01.040518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.040488    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"]
Nov 02 07:44:01.045659 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.045634    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6cc3f7\""
Nov 02 07:44:01.045801 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.045735    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6cc3f7-dockercfg-7sdmx\""
Nov 02 07:44:01.088532 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.088500    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"]
Nov 02 07:44:01.089835 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.089776    4898 cpu_manager.go:414] "containerMap: removing container" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy"
Nov 02 07:44:01.090001 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.089983    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82360126-e2f7-4bf0-9abf-3c82168fb797" containerName="istio-proxy"
Nov 02 07:44:01.096277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.096251    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"
Nov 02 07:44:01.125912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.123321    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67xj4\" (UniqueName: \"kubernetes.io/projected/1b947b1b-478c-4661-908f-7e04c60bd5e3-kube-api-access-67xj4\") pod \"payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx\" (UID: \"1b947b1b-478c-4661-908f-7e04c60bd5e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"
Nov 02 07:44:01.156950 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.131076    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b947b1b-478c-4661-908f-7e04c60bd5e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx\" (UID: \"1b947b1b-478c-4661-908f-7e04c60bd5e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"
Nov 02 07:44:01.295317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.286858    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf239166-12f0-4cdf-a95f-88df4437dc1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk\" (UID: \"cf239166-12f0-4cdf-a95f-88df4437dc1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"
Nov 02 07:44:01.295317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.286977    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmw96\" (UniqueName: \"kubernetes.io/projected/cf239166-12f0-4cdf-a95f-88df4437dc1a-kube-api-access-jmw96\") pod \"payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk\" (UID: \"cf239166-12f0-4cdf-a95f-88df4437dc1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"
Nov 02 07:44:01.295317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.287013    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67xj4\" (UniqueName: \"kubernetes.io/projected/1b947b1b-478c-4661-908f-7e04c60bd5e3-kube-api-access-67xj4\") pod \"payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx\" (UID: \"1b947b1b-478c-4661-908f-7e04c60bd5e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"
Nov 02 07:44:01.295317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.287190    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b947b1b-478c-4661-908f-7e04c60bd5e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx\" (UID: \"1b947b1b-478c-4661-908f-7e04c60bd5e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"
Nov 02 07:44:01.295317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.288119    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b947b1b-478c-4661-908f-7e04c60bd5e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx\" (UID: \"1b947b1b-478c-4661-908f-7e04c60bd5e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"
Nov 02 07:44:01.324003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.323965    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"]
Nov 02 07:44:01.340564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.340519    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67xj4\" (UniqueName: \"kubernetes.io/projected/1b947b1b-478c-4661-908f-7e04c60bd5e3-kube-api-access-67xj4\") pod \"payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx\" (UID: \"1b947b1b-478c-4661-908f-7e04c60bd5e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"
Nov 02 07:44:01.388528 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.388488    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf239166-12f0-4cdf-a95f-88df4437dc1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk\" (UID: \"cf239166-12f0-4cdf-a95f-88df4437dc1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"
Nov 02 07:44:01.388722 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.388578    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmw96\" (UniqueName: \"kubernetes.io/projected/cf239166-12f0-4cdf-a95f-88df4437dc1a-kube-api-access-jmw96\") pod \"payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk\" (UID: \"cf239166-12f0-4cdf-a95f-88df4437dc1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"
Nov 02 07:44:01.389355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.389331    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf239166-12f0-4cdf-a95f-88df4437dc1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk\" (UID: \"cf239166-12f0-4cdf-a95f-88df4437dc1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"
Nov 02 07:44:01.424141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.424109    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmw96\" (UniqueName: \"kubernetes.io/projected/cf239166-12f0-4cdf-a95f-88df4437dc1a-kube-api-access-jmw96\") pod \"payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk\" (UID: \"cf239166-12f0-4cdf-a95f-88df4437dc1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"
Nov 02 07:44:01.611817 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.611726    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"
Nov 02 07:44:01.629809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.629784    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"
Nov 02 07:44:01.981669 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.981640    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"]
Nov 02 07:44:02.007291 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:01.998062    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"]
Nov 02 07:44:02.740024 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:02.738674    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx" event={"ID":"1b947b1b-478c-4661-908f-7e04c60bd5e3","Type":"ContainerStarted","Data":"31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8"}
Nov 02 07:44:02.740024 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:02.738727    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx" event={"ID":"1b947b1b-478c-4661-908f-7e04c60bd5e3","Type":"ContainerStarted","Data":"0b4feae1f1993bec9d6f5b597cc82a2c1ad05f58cba558e90890bfdf1bff26c3"}
Nov 02 07:44:02.740958 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:02.740572    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"
Nov 02 07:44:02.743711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:02.743070    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk" event={"ID":"cf239166-12f0-4cdf-a95f-88df4437dc1a","Type":"ContainerStarted","Data":"f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76"}
Nov 02 07:44:02.743711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:02.743112    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk" event={"ID":"cf239166-12f0-4cdf-a95f-88df4437dc1a","Type":"ContainerStarted","Data":"89986f74f6884d9317f44260266691e836810fd5543dec2b9acc083a46d58bc2"}
Nov 02 07:44:02.743711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:02.743664    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"
Nov 02 07:44:02.785479 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:02.785428    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx" podStartSLOduration=2.785416728 podStartE2EDuration="2.785416728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:02.783374316 +0000 UTC m=+1193.473701280" watchObservedRunningTime="2026-11-02 07:44:02.785416728 +0000 UTC m=+1193.475743690"
Nov 02 07:44:02.832541 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:02.832489    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk" podStartSLOduration=1.8324735479999998 podStartE2EDuration="1.832473548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:02.828794968 +0000 UTC m=+1193.519121943" watchObservedRunningTime="2026-11-02 07:44:02.832473548 +0000 UTC m=+1193.522800511"
Nov 02 07:44:04.034976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:04.034944    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"]
Nov 02 07:44:04.046147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:04.046118    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9febe4d7-nns7k"]
Nov 02 07:44:05.467157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:05.467122    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550bc00f-8763-442c-b9af-15abeba8f4c8" path="/var/lib/kubelet/pods/550bc00f-8763-442c-b9af-15abeba8f4c8/volumes"
Nov 02 07:44:09.529343 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.529306    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"]
Nov 02 07:44:09.536352 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.536325    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"
Nov 02 07:44:09.566550 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.566524    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"]
Nov 02 07:44:09.570863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.570842    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9d12c2\""
Nov 02 07:44:09.571114 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.571030    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9d12c2-dockercfg-pkkhn\""
Nov 02 07:44:09.595411 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.595380    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"]
Nov 02 07:44:09.601159 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.601127    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"
Nov 02 07:44:09.635785 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.635753    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"]
Nov 02 07:44:09.649582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.649539    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2sd8\" (UniqueName: \"kubernetes.io/projected/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-kube-api-access-t2sd8\") pod \"payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865\" (UID: \"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"
Nov 02 07:44:09.650044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.649992    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865\" (UID: \"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"
Nov 02 07:44:09.797965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.766619    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2sd8\" (UniqueName: \"kubernetes.io/projected/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-kube-api-access-t2sd8\") pod \"payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865\" (UID: \"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"
Nov 02 07:44:09.804063 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.798183    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1663b127-9ab2-45b6-afc5-78bb5236aa47-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm\" (UID: \"1663b127-9ab2-45b6-afc5-78bb5236aa47\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"
Nov 02 07:44:09.826122 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.826085    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxkkl\" (UniqueName: \"kubernetes.io/projected/1663b127-9ab2-45b6-afc5-78bb5236aa47-kube-api-access-vxkkl\") pod \"payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm\" (UID: \"1663b127-9ab2-45b6-afc5-78bb5236aa47\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"
Nov 02 07:44:09.826632 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.826604    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865\" (UID: \"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"
Nov 02 07:44:09.835679 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.833440    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2sd8\" (UniqueName: \"kubernetes.io/projected/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-kube-api-access-t2sd8\") pod \"payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865\" (UID: \"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"
Nov 02 07:44:09.849833 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.849038    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865\" (UID: \"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"
Nov 02 07:44:09.862150 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.861973    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"
Nov 02 07:44:09.904361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.904272    4898 scope.go:122] "RemoveContainer" containerID="745455174c9b1f34d264fbdc85ac3f9f33ff4f929f4905d745d9dc76bb57c1e0"
Nov 02 07:44:09.929373 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.929336    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1663b127-9ab2-45b6-afc5-78bb5236aa47-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm\" (UID: \"1663b127-9ab2-45b6-afc5-78bb5236aa47\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"
Nov 02 07:44:09.936783 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.936760    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxkkl\" (UniqueName: \"kubernetes.io/projected/1663b127-9ab2-45b6-afc5-78bb5236aa47-kube-api-access-vxkkl\") pod \"payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm\" (UID: \"1663b127-9ab2-45b6-afc5-78bb5236aa47\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"
Nov 02 07:44:09.937583 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.934732    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1663b127-9ab2-45b6-afc5-78bb5236aa47-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm\" (UID: \"1663b127-9ab2-45b6-afc5-78bb5236aa47\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"
Nov 02 07:44:09.966943 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.966871    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxkkl\" (UniqueName: \"kubernetes.io/projected/1663b127-9ab2-45b6-afc5-78bb5236aa47-kube-api-access-vxkkl\") pod \"payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm\" (UID: \"1663b127-9ab2-45b6-afc5-78bb5236aa47\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"
Nov 02 07:44:09.975865 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.975839    4898 scope.go:122] "RemoveContainer" containerID="ecfa67f733a75e6b602965e4fdd437d2c04ddce1c6b4d617886d77c905b79122"
Nov 02 07:44:09.986182 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:44:09.985987    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745455174c9b1f34d264fbdc85ac3f9f33ff4f929f4905d745d9dc76bb57c1e0\": container with ID starting with 745455174c9b1f34d264fbdc85ac3f9f33ff4f929f4905d745d9dc76bb57c1e0 not found: ID does not exist" containerID="745455174c9b1f34d264fbdc85ac3f9f33ff4f929f4905d745d9dc76bb57c1e0"
Nov 02 07:44:09.986182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.986034    4898 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"745455174c9b1f34d264fbdc85ac3f9f33ff4f929f4905d745d9dc76bb57c1e0\": container with ID starting with 745455174c9b1f34d264fbdc85ac3f9f33ff4f929f4905d745d9dc76bb57c1e0 not found: ID does not exist"
Nov 02 07:44:09.987171 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:44:09.987141    4898 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"745455174c9b1f34d264fbdc85ac3f9f33ff4f929f4905d745d9dc76bb57c1e0\": container with ID starting with 745455174c9b1f34d264fbdc85ac3f9f33ff4f929f4905d745d9dc76bb57c1e0 not found: ID does not exist"
Nov 02 07:44:09.998417 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:09.998394    4898 scope.go:122] "RemoveContainer" containerID="35a43a480d3311c55e544b4f96ae79491237d5769787c618644717404bfc4273"
Nov 02 07:44:10.020242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:10.020219    4898 scope.go:122] "RemoveContainer" containerID="0cd584d44984a09034f254c2e3cd99963042c00b0335cd00331475bd655c18a8"
Nov 02 07:44:10.049222 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:10.049197    4898 scope.go:122] "RemoveContainer" containerID="b93fd397c59656a24e5b9c12658fd4e02872f6eec4704b70bb9c61295d3e4818"
Nov 02 07:44:10.221394 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:10.221361    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"
Nov 02 07:44:10.274922 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:44:10.274496    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice/crio-c1d4419264f28859fb8b6b7ea3854ea980e6828b3853ba6ea6febede0623f60b WatchSource:0}: Error finding container c1d4419264f28859fb8b6b7ea3854ea980e6828b3853ba6ea6febede0623f60b: Status 404 returned error can't find the container with id c1d4419264f28859fb8b6b7ea3854ea980e6828b3853ba6ea6febede0623f60b
Nov 02 07:44:10.277686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:10.277655    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"]
Nov 02 07:44:10.541297 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:44:10.541265    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1663b127_9ab2_45b6_afc5_78bb5236aa47.slice/crio-65e97183d744fa4cf9cc3ae4a18fc7a3f0e6d98fd253d1ef708992041abe9d7e WatchSource:0}: Error finding container 65e97183d744fa4cf9cc3ae4a18fc7a3f0e6d98fd253d1ef708992041abe9d7e: Status 404 returned error can't find the container with id 65e97183d744fa4cf9cc3ae4a18fc7a3f0e6d98fd253d1ef708992041abe9d7e
Nov 02 07:44:10.543737 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:10.543712    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"]
Nov 02 07:44:10.914141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:10.914100    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm" event={"ID":"1663b127-9ab2-45b6-afc5-78bb5236aa47","Type":"ContainerStarted","Data":"5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0"}
Nov 02 07:44:10.914343 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:10.914171    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"
Nov 02 07:44:10.914343 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:10.914194    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm" event={"ID":"1663b127-9ab2-45b6-afc5-78bb5236aa47","Type":"ContainerStarted","Data":"65e97183d744fa4cf9cc3ae4a18fc7a3f0e6d98fd253d1ef708992041abe9d7e"}
Nov 02 07:44:10.916257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:10.916229    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865" event={"ID":"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf","Type":"ContainerStarted","Data":"8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f"}
Nov 02 07:44:10.916393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:10.916276    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"
Nov 02 07:44:10.916393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:10.916292    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865" event={"ID":"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf","Type":"ContainerStarted","Data":"c1d4419264f28859fb8b6b7ea3854ea980e6828b3853ba6ea6febede0623f60b"}
Nov 02 07:44:10.980857 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:10.980806    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm" podStartSLOduration=1.980791021 podStartE2EDuration="1.980791021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:10.97651869 +0000 UTC m=+1201.666845673" watchObservedRunningTime="2026-11-02 07:44:10.980791021 +0000 UTC m=+1201.671117985"
Nov 02 07:44:11.046003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:11.045935    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865" podStartSLOduration=2.045916389 podStartE2EDuration="2.045916389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:11.0394672 +0000 UTC m=+1201.729794167" watchObservedRunningTime="2026-11-02 07:44:11.045916389 +0000 UTC m=+1201.736243352"
Nov 02 07:44:13.756342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:13.756305    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"
Nov 02 07:44:14.767456 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:14.767427    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"
Nov 02 07:44:14.998782 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:44:14.998746    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:44:18.114788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:18.114755    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"]
Nov 02 07:44:18.138728 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:18.138699    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fabe65f5-gv6mb"]
Nov 02 07:44:19.466797 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:19.466762    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="328ac24d-ccf6-4faf-93d7-42aa4238923e" path="/var/lib/kubelet/pods/328ac24d-ccf6-4faf-93d7-42aa4238923e/volumes"
Nov 02 07:44:20.128740 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.128705    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"]
Nov 02 07:44:20.145203 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.143642    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"
Nov 02 07:44:20.166539 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.158684    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"]
Nov 02 07:44:20.166539 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.159970    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7e07c6\""
Nov 02 07:44:20.221908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.220299    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7e07c6-dockercfg-z5rks\""
Nov 02 07:44:20.251022 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.250984    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2xsn\" (UniqueName: \"kubernetes.io/projected/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-kube-api-access-s2xsn\") pod \"payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn\" (UID: \"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"
Nov 02 07:44:20.251332 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.251111    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn\" (UID: \"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"
Nov 02 07:44:20.356997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.354667    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2xsn\" (UniqueName: \"kubernetes.io/projected/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-kube-api-access-s2xsn\") pod \"payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn\" (UID: \"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"
Nov 02 07:44:20.356997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.354792    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn\" (UID: \"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"
Nov 02 07:44:20.356997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.355847    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn\" (UID: \"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"
Nov 02 07:44:20.394966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.394616    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2xsn\" (UniqueName: \"kubernetes.io/projected/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-kube-api-access-s2xsn\") pod \"payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn\" (UID: \"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"
Nov 02 07:44:20.489488 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.489451    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"
Nov 02 07:44:20.836254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:20.831457    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"]
Nov 02 07:44:21.126570 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:21.126473    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn" event={"ID":"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1","Type":"ContainerStarted","Data":"d15084725ced2dbc208aeb8259e2c84e6ded6c28ea86ccc905d2578ee042140c"}
Nov 02 07:44:21.928838 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:21.928616    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"
Nov 02 07:44:21.929728 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:21.929704    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"
Nov 02 07:44:22.615911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.614915    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"]
Nov 02 07:44:22.620697 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.619924    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.657094 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.657068    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5a032f-openshift-default-dockercfg-rj4kc\""
Nov 02 07:44:22.665181 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.665158    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"]
Nov 02 07:44:22.682910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.672877    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-credential-socket\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.682910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.675648    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-token\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.682910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.675779    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-certs\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.682910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.675834    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkhp6\" (UniqueName: \"kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-kube-api-access-fkhp6\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.682910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.675903    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-envoy\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.682910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.675960    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-socket\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.682910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.675995    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbf728-4e49-4574-9bc8-1d53e3356957-istiod-ca-cert\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.682910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.676023    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-podinfo\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.682910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.676066    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-data\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.783908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.783854    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-socket\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.786345 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.784505    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbf728-4e49-4574-9bc8-1d53e3356957-istiod-ca-cert\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.786345 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.784429    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-socket\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.786345 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.785839    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-podinfo\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.786647 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.786559    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-data\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.787233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.786902    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-data\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.787233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.787070    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-credential-socket\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.787233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.787137    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbf728-4e49-4574-9bc8-1d53e3356957-istiod-ca-cert\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.787458 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.787360    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-credential-socket\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.787949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.787863    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-token\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.788058 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.788038    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-certs\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.788422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.788399    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-certs\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.788505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.788491    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkhp6\" (UniqueName: \"kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-kube-api-access-fkhp6\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.788564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.788549    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-envoy\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.794176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.794152    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-podinfo\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.800098 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.799184    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-envoy\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.815194 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.815165    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkhp6\" (UniqueName: \"kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-kube-api-access-fkhp6\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.815194 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.815192    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-token\") pod \"e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") " pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:22.961482 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:22.961431    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:23.368997 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:44:23.368966    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fbf728_4e49_4574_9bc8_1d53e3356957.slice/crio-cd5e9e07b779f9a69cdb8dd504d04c256eef5d9e6c68b8a1d674ec2e99288c80 WatchSource:0}: Error finding container cd5e9e07b779f9a69cdb8dd504d04c256eef5d9e6c68b8a1d674ec2e99288c80: Status 404 returned error can't find the container with id cd5e9e07b779f9a69cdb8dd504d04c256eef5d9e6c68b8a1d674ec2e99288c80
Nov 02 07:44:23.370652 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:23.370620    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"]
Nov 02 07:44:23.372927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:23.372868    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:23.373046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:23.372971    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:23.373046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:23.373017    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:24.185051 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:24.185011    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" event={"ID":"38fbf728-4e49-4574-9bc8-1d53e3356957","Type":"ContainerStarted","Data":"bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7"}
Nov 02 07:44:24.185509 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:24.185086    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" event={"ID":"38fbf728-4e49-4574-9bc8-1d53e3356957","Type":"ContainerStarted","Data":"cd5e9e07b779f9a69cdb8dd504d04c256eef5d9e6c68b8a1d674ec2e99288c80"}
Nov 02 07:44:24.186986 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:24.186935    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn" event={"ID":"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1","Type":"ContainerStarted","Data":"1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a"}
Nov 02 07:44:24.187108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:24.186994    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"
Nov 02 07:44:24.261193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:24.261083    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" podStartSLOduration=2.261068919 podStartE2EDuration="2.261068919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:24.245659314 +0000 UTC m=+1214.935986309" watchObservedRunningTime="2026-11-02 07:44:24.261068919 +0000 UTC m=+1214.951395882"
Nov 02 07:44:24.292109 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:24.292055    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn" podStartSLOduration=1.124673897 podStartE2EDuration="4.292038269s" totalImagesPullingTime="3.167364372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:24.287027383 +0000 UTC m=+1214.977354346" watchObservedRunningTime="2026-11-02 07:44:24.292038269 +0000 UTC m=+1214.982365233"
Nov 02 07:44:24.961808 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:24.961770    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:24.968534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:24.968509    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:25.200912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:25.199918    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:25.201362 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:25.201181    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:44:29.039758 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:29.039216    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"]
Nov 02 07:44:29.053323 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:29.053287    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42fa15d-7nljl"]
Nov 02 07:44:29.465758 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:29.465726    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9877ef1-2f01-46bb-af62-6753549ce0ea" path="/var/lib/kubelet/pods/c9877ef1-2f01-46bb-af62-6753549ce0ea/volumes"
Nov 02 07:44:32.122321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.122287    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"]
Nov 02 07:44:32.127354 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.127330    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.150578 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.149680    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-523de0-openshift-default-dockercfg-ptktx\""
Nov 02 07:44:32.175595 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.175566    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"]
Nov 02 07:44:32.190294 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.190263    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-data\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.190456 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.190325    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.190456 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.190386    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-token\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.190456 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.190439    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-envoy\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.190589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.190537    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-certs\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.190589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.190572    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-podinfo\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.190690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.190638    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-credential-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.190842 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.190827    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istiod-ca-cert\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.190999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.190979    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkddr\" (UniqueName: \"kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-kube-api-access-zkddr\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.292148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.292115    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkddr\" (UniqueName: \"kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-kube-api-access-zkddr\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.292346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.292211    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-data\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.292346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.292255    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.292346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.292284    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-token\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.292346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.292318    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-envoy\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.292565 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.292353    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-certs\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.292565 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.292379    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-podinfo\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.292565 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.292412    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-credential-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.292725 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.292620    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istiod-ca-cert\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.293840 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.293531    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istiod-ca-cert\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.293840 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.293662    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.293840 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.293791    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-data\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.294687 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.294636    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-credential-socket\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.294808 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.294702    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-certs\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.296925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.296404    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-podinfo\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.298032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.297365    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-envoy\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.321427 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.321346    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-token\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.321589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.321480    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkddr\" (UniqueName: \"kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-kube-api-access-zkddr\") pod \"e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") " pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.443019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.442987    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:32.816364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.816323    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:32.816483 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.816396    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:32.816483 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.816429    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:44:32.818366 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:32.818342    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"]
Nov 02 07:44:33.280809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:33.280761    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz" event={"ID":"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226","Type":"ContainerStarted","Data":"672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973"}
Nov 02 07:44:33.281249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:33.280848    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz" event={"ID":"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226","Type":"ContainerStarted","Data":"1dee96bebf5dd25d2ce2f13075d86b25e4162cf09f25a2ef13882d3d5649e497"}
Nov 02 07:44:33.357939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:33.355508    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz" podStartSLOduration=1.355489329 podStartE2EDuration="1.355489329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:44:33.339407388 +0000 UTC m=+1224.029734352" watchObservedRunningTime="2026-11-02 07:44:33.355489329 +0000 UTC m=+1224.045816292"
Nov 02 07:44:33.443976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:33.443939    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:33.445169 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:33.445138    4898 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 07:44:33.445287 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:33.445199    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz" podUID="b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 07:44:34.443682 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:34.443645    4898 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 07:44:34.444116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:34.443699    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz" podUID="b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 07:44:35.053167 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:35.053126    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"]
Nov 02 07:44:35.085878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:35.085844    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6e2994-wpdch"]
Nov 02 07:44:35.202344 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:35.202315    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"
Nov 02 07:44:35.444209 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:35.444155    4898 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 07:44:35.444708 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:35.444228    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz" podUID="b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 07:44:35.489042 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:35.489007    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc9f27f0-ab89-41d1-bb4a-0f620923b4d5" path="/var/lib/kubelet/pods/cc9f27f0-ab89-41d1-bb4a-0f620923b4d5/volumes"
Nov 02 07:44:36.444176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:36.444134    4898 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 07:44:36.444375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:36.444213    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz" podUID="b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 07:44:37.444017 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:37.443977    4898 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 07:44:37.444243 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:37.444040    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz" podUID="b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 07:44:38.444155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:38.444113    4898 patch_prober.go:29] interesting pod/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 07:44:38.444686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:38.444203    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz" podUID="b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 07:44:39.451591 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:39.451522    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:39.452924 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:39.452713    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:39.453272 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:39.453241    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:44:40.084130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:40.084092    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"]
Nov 02 07:44:40.095946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:40.095917    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6556b310-p7mjd"]
Nov 02 07:44:41.481217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:41.481176    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a67f57-f503-4945-81e9-a4d9feadb29a" path="/var/lib/kubelet/pods/69a67f57-f503-4945-81e9-a4d9feadb29a/volumes"
Nov 02 07:44:44.992673 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:44:44.992629    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:44:46.034752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:46.034715    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"]
Nov 02 07:44:46.049651 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:46.049614    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f169cbee-clgtz"]
Nov 02 07:44:47.466495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:47.466453    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="737594a7-28b1-490a-ad31-7e7ffbf30459" path="/var/lib/kubelet/pods/737594a7-28b1-490a-ad31-7e7ffbf30459/volumes"
Nov 02 07:44:50.035758 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:50.035727    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"]
Nov 02 07:44:50.132937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:50.132873    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"]
Nov 02 07:44:51.068027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:51.067989    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"]
Nov 02 07:44:51.104000 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:51.103933    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc51541b-ktltt"]
Nov 02 07:44:51.468625 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:51.468582    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f40e3b-4a74-473b-889b-f7d77c2dd65f" path="/var/lib/kubelet/pods/a8f40e3b-4a74-473b-889b-f7d77c2dd65f/volumes"
Nov 02 07:44:52.035901 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:52.035867    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"]
Nov 02 07:44:52.047541 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:52.047510    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f501c6-g5jkd"]
Nov 02 07:44:53.470052 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:53.470019    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36376945-8bce-4163-8129-c933996144dc" path="/var/lib/kubelet/pods/36376945-8bce-4163-8129-c933996144dc/volumes"
Nov 02 07:44:55.045269 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.043538    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk" podUID="cf239166-12f0-4cdf-a95f-88df4437dc1a" containerName="payload-processing" containerID="cri-o://f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76" gracePeriod=25
Nov 02 07:44:55.133721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.133654    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx" podUID="1b947b1b-478c-4661-908f-7e04c60bd5e3" containerName="payload-pre-processing" containerID="cri-o://31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8" gracePeriod=25
Nov 02 07:44:55.428165 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.428135    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"
Nov 02 07:44:55.471904 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.471869    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"
Nov 02 07:44:55.504366 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.504330    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf239166-12f0-4cdf-a95f-88df4437dc1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf239166-12f0-4cdf-a95f-88df4437dc1a-plugins-config-volume\") pod \"cf239166-12f0-4cdf-a95f-88df4437dc1a\" (UID: \"cf239166-12f0-4cdf-a95f-88df4437dc1a\") "
Nov 02 07:44:55.504532 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.504492    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf239166-12f0-4cdf-a95f-88df4437dc1a-kube-api-access-jmw96\" (UniqueName: \"kubernetes.io/projected/cf239166-12f0-4cdf-a95f-88df4437dc1a-kube-api-access-jmw96\") pod \"cf239166-12f0-4cdf-a95f-88df4437dc1a\" (UID: \"cf239166-12f0-4cdf-a95f-88df4437dc1a\") "
Nov 02 07:44:55.504805 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.504773    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf239166-12f0-4cdf-a95f-88df4437dc1a-plugins-config-volume" pod "cf239166-12f0-4cdf-a95f-88df4437dc1a" (UID: "cf239166-12f0-4cdf-a95f-88df4437dc1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:44:55.505349 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.505323    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf239166-12f0-4cdf-a95f-88df4437dc1a-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:44:55.506827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.506803    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf239166-12f0-4cdf-a95f-88df4437dc1a-kube-api-access-jmw96" pod "cf239166-12f0-4cdf-a95f-88df4437dc1a" (UID: "cf239166-12f0-4cdf-a95f-88df4437dc1a"). InnerVolumeSpecName "kube-api-access-jmw96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:44:55.606164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.606082    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b947b1b-478c-4661-908f-7e04c60bd5e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b947b1b-478c-4661-908f-7e04c60bd5e3-plugins-config-volume\") pod \"1b947b1b-478c-4661-908f-7e04c60bd5e3\" (UID: \"1b947b1b-478c-4661-908f-7e04c60bd5e3\") "
Nov 02 07:44:55.606403 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.606388    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b947b1b-478c-4661-908f-7e04c60bd5e3-kube-api-access-67xj4\" (UniqueName: \"kubernetes.io/projected/1b947b1b-478c-4661-908f-7e04c60bd5e3-kube-api-access-67xj4\") pod \"1b947b1b-478c-4661-908f-7e04c60bd5e3\" (UID: \"1b947b1b-478c-4661-908f-7e04c60bd5e3\") "
Nov 02 07:44:55.606457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.606411    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b947b1b-478c-4661-908f-7e04c60bd5e3-plugins-config-volume" pod "1b947b1b-478c-4661-908f-7e04c60bd5e3" (UID: "1b947b1b-478c-4661-908f-7e04c60bd5e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:44:55.606813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.606798    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmw96\" (UniqueName: \"kubernetes.io/projected/cf239166-12f0-4cdf-a95f-88df4437dc1a-kube-api-access-jmw96\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:44:55.606813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.606813    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b947b1b-478c-4661-908f-7e04c60bd5e3-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:44:55.608679 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.608658    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b947b1b-478c-4661-908f-7e04c60bd5e3-kube-api-access-67xj4" pod "1b947b1b-478c-4661-908f-7e04c60bd5e3" (UID: "1b947b1b-478c-4661-908f-7e04c60bd5e3"). InnerVolumeSpecName "kube-api-access-67xj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:44:55.708061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.708029    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67xj4\" (UniqueName: \"kubernetes.io/projected/1b947b1b-478c-4661-908f-7e04c60bd5e3-kube-api-access-67xj4\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:44:55.841824 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.841786    4898 generic.go:356] "Generic (PLEG): container finished" podID="cf239166-12f0-4cdf-a95f-88df4437dc1a" containerID="f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76" exitCode=0
Nov 02 07:44:55.842048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.841833    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"
Nov 02 07:44:55.842048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.841872    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk" event={"ID":"cf239166-12f0-4cdf-a95f-88df4437dc1a","Type":"ContainerDied","Data":"f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76"}
Nov 02 07:44:55.842048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.841959    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk" event={"ID":"cf239166-12f0-4cdf-a95f-88df4437dc1a","Type":"ContainerDied","Data":"89986f74f6884d9317f44260266691e836810fd5543dec2b9acc083a46d58bc2"}
Nov 02 07:44:55.842048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.841992    4898 scope.go:122] "RemoveContainer" containerID="f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76"
Nov 02 07:44:55.843584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.843550    4898 generic.go:356] "Generic (PLEG): container finished" podID="1b947b1b-478c-4661-908f-7e04c60bd5e3" containerID="31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8" exitCode=0
Nov 02 07:44:55.843730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.843597    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"
Nov 02 07:44:55.843730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.843650    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx" event={"ID":"1b947b1b-478c-4661-908f-7e04c60bd5e3","Type":"ContainerDied","Data":"31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8"}
Nov 02 07:44:55.843730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.843705    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx" event={"ID":"1b947b1b-478c-4661-908f-7e04c60bd5e3","Type":"ContainerDied","Data":"0b4feae1f1993bec9d6f5b597cc82a2c1ad05f58cba558e90890bfdf1bff26c3"}
Nov 02 07:44:55.855957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.855934    4898 scope.go:122] "RemoveContainer" containerID="f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76"
Nov 02 07:44:55.856347 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:44:55.856301    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76\": container with ID starting with f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76 not found: ID does not exist" containerID="f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76"
Nov 02 07:44:55.856436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.856333    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76"} err="failed to get container status \"f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76\": rpc error: code = NotFound desc = could not find container \"f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76\": container with ID starting with f23454a88ef680d0efd97e08d8fae5409905a3b504bdddfde64bffefb5e97a76 not found: ID does not exist"
Nov 02 07:44:55.856436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.856359    4898 scope.go:122] "RemoveContainer" containerID="31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8"
Nov 02 07:44:55.868840 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.868817    4898 scope.go:122] "RemoveContainer" containerID="31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8"
Nov 02 07:44:55.869184 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:44:55.869155    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8\": container with ID starting with 31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8 not found: ID does not exist" containerID="31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8"
Nov 02 07:44:55.869259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.869188    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8"} err="failed to get container status \"31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8\": rpc error: code = NotFound desc = could not find container \"31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8\": container with ID starting with 31f8508c58140357afdfa2b5fa057460ae1190f6b64bd7b5799386edb493daa8 not found: ID does not exist"
Nov 02 07:44:55.901065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.901032    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"]
Nov 02 07:44:55.908958 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.908932    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cc3f7-854f9d8497-frzzk"]
Nov 02 07:44:55.943629 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.943599    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"]
Nov 02 07:44:55.951404 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:55.951374    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-5f46dfcf79-hvhbx"]
Nov 02 07:44:56.045530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:56.045487    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"]
Nov 02 07:44:56.050875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:56.050843    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9034bc-k8xq2"]
Nov 02 07:44:57.465996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:57.465966    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b947b1b-478c-4661-908f-7e04c60bd5e3" path="/var/lib/kubelet/pods/1b947b1b-478c-4661-908f-7e04c60bd5e3/volumes"
Nov 02 07:44:57.466449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:57.466434    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42c627f6-410b-4f79-b096-f74f6ac6dc3a" path="/var/lib/kubelet/pods/42c627f6-410b-4f79-b096-f74f6ac6dc3a/volumes"
Nov 02 07:44:57.466817 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:44:57.466805    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf239166-12f0-4cdf-a95f-88df4437dc1a" path="/var/lib/kubelet/pods/cf239166-12f0-4cdf-a95f-88df4437dc1a/volumes"
Nov 02 07:45:00.157861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.157821    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"]
Nov 02 07:45:00.158625 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.158605    4898 cpu_manager.go:414] "containerMap: removing container" podUID="cf239166-12f0-4cdf-a95f-88df4437dc1a" containerName="payload-processing"
Nov 02 07:45:00.158625 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.158624    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf239166-12f0-4cdf-a95f-88df4437dc1a" containerName="payload-processing"
Nov 02 07:45:00.158762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.158668    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1b947b1b-478c-4661-908f-7e04c60bd5e3" containerName="payload-pre-processing"
Nov 02 07:45:00.158762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.158674    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b947b1b-478c-4661-908f-7e04c60bd5e3" containerName="payload-pre-processing"
Nov 02 07:45:00.158849 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.158775    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b947b1b-478c-4661-908f-7e04c60bd5e3" containerName="payload-pre-processing"
Nov 02 07:45:00.158849 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.158787    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf239166-12f0-4cdf-a95f-88df4437dc1a" containerName="payload-processing"
Nov 02 07:45:00.162971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.162951    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"
Nov 02 07:45:00.163066 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.162963    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"]
Nov 02 07:45:00.167702 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.167681    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"
Nov 02 07:45:00.169294 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.169272    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"]
Nov 02 07:45:00.173994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.173978    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"
Nov 02 07:45:00.175579 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.175561    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"]
Nov 02 07:45:00.183982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.183956    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:45:00.191390 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.191207    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:45:00.191390 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.191362    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:45:00.199152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.199127    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"]
Nov 02 07:45:00.200716 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.200695    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"
Nov 02 07:45:00.205038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.205016    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"]
Nov 02 07:45:00.205150 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.205058    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"]
Nov 02 07:45:00.205150 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.205099    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"]
Nov 02 07:45:00.205263 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.205205    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"
Nov 02 07:45:00.210743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.210723    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"]
Nov 02 07:45:00.216104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.216081    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv95h\" (UniqueName: \"kubernetes.io/projected/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-kube-api-access-jv95h\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824\" (UID: \"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"
Nov 02 07:45:00.216384 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.216358    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82934468-ad39-4323-91c5-37163776131f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4\" (UID: \"82934468-ad39-4323-91c5-37163776131f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"
Nov 02 07:45:00.216617 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.216597    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxcg\" (UniqueName: \"kubernetes.io/projected/82934468-ad39-4323-91c5-37163776131f-kube-api-access-xhxcg\") pod \"maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4\" (UID: \"82934468-ad39-4323-91c5-37163776131f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"
Nov 02 07:45:00.216746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.216730    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824\" (UID: \"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"
Nov 02 07:45:00.216981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.216963    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glm9g\" (UniqueName: \"kubernetes.io/projected/fdc34509-c942-498b-9324-ae77e0b22c03-kube-api-access-glm9g\") pod \"maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7\" (UID: \"fdc34509-c942-498b-9324-ae77e0b22c03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"
Nov 02 07:45:00.217128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.217113    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc34509-c942-498b-9324-ae77e0b22c03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7\" (UID: \"fdc34509-c942-498b-9324-ae77e0b22c03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"
Nov 02 07:45:00.217269 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.217250    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4a75ea-2924-43ca-af92-1cf9cd51def8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9\" (UID: \"bf4a75ea-2924-43ca-af92-1cf9cd51def8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"
Nov 02 07:45:00.218490 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.218470    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmxws\" (UniqueName: \"kubernetes.io/projected/b1e9db58-0abc-488f-8002-49ef49d20cac-kube-api-access-rmxws\") pod \"maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t\" (UID: \"b1e9db58-0abc-488f-8002-49ef49d20cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"
Nov 02 07:45:00.218648 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.218629    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psxdj\" (UniqueName: \"kubernetes.io/projected/bf4a75ea-2924-43ca-af92-1cf9cd51def8-kube-api-access-psxdj\") pod \"maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9\" (UID: \"bf4a75ea-2924-43ca-af92-1cf9cd51def8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"
Nov 02 07:45:00.218768 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.218752    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e9db58-0abc-488f-8002-49ef49d20cac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t\" (UID: \"b1e9db58-0abc-488f-8002-49ef49d20cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"
Nov 02 07:45:00.219082 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.217499    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"]
Nov 02 07:45:00.220675 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.217565    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"
Nov 02 07:45:00.239912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.238689    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"]
Nov 02 07:45:00.241869 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.241844    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"
Nov 02 07:45:00.245190 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.245171    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"]
Nov 02 07:45:00.245364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.245350    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"]
Nov 02 07:45:00.245480 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.245471    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"]
Nov 02 07:45:00.246810 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.246794    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"
Nov 02 07:45:00.249077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.249057    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"]
Nov 02 07:45:00.259136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.259114    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"]
Nov 02 07:45:00.320067 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320027    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234371f3-b3bf-4203-9dd3-d9dc57e666d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf\" (UID: \"234371f3-b3bf-4203-9dd3-d9dc57e666d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"
Nov 02 07:45:00.320261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320094    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv95h\" (UniqueName: \"kubernetes.io/projected/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-kube-api-access-jv95h\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824\" (UID: \"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"
Nov 02 07:45:00.320261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320153    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35036d5-6707-415a-ae82-ccd088c58cc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d\" (UID: \"b35036d5-6707-415a-ae82-ccd088c58cc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"
Nov 02 07:45:00.320261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320226    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82934468-ad39-4323-91c5-37163776131f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4\" (UID: \"82934468-ad39-4323-91c5-37163776131f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"
Nov 02 07:45:00.320422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320274    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c22c\" (UniqueName: \"kubernetes.io/projected/b35036d5-6707-415a-ae82-ccd088c58cc7-kube-api-access-4c22c\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d\" (UID: \"b35036d5-6707-415a-ae82-ccd088c58cc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"
Nov 02 07:45:00.320422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320354    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxcg\" (UniqueName: \"kubernetes.io/projected/82934468-ad39-4323-91c5-37163776131f-kube-api-access-xhxcg\") pod \"maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4\" (UID: \"82934468-ad39-4323-91c5-37163776131f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"
Nov 02 07:45:00.320422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320376    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824\" (UID: \"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"
Nov 02 07:45:00.320584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320451    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/626b6a54-25a8-4149-86f8-b2495d96b931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp\" (UID: \"626b6a54-25a8-4149-86f8-b2495d96b931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"
Nov 02 07:45:00.320584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320516    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glm9g\" (UniqueName: \"kubernetes.io/projected/fdc34509-c942-498b-9324-ae77e0b22c03-kube-api-access-glm9g\") pod \"maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7\" (UID: \"fdc34509-c942-498b-9324-ae77e0b22c03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"
Nov 02 07:45:00.320694 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320572    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc34509-c942-498b-9324-ae77e0b22c03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7\" (UID: \"fdc34509-c942-498b-9324-ae77e0b22c03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"
Nov 02 07:45:00.320694 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320616    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjth\" (UniqueName: \"kubernetes.io/projected/626b6a54-25a8-4149-86f8-b2495d96b931-kube-api-access-nzjth\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp\" (UID: \"626b6a54-25a8-4149-86f8-b2495d96b931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"
Nov 02 07:45:00.320694 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320675    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4a75ea-2924-43ca-af92-1cf9cd51def8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9\" (UID: \"bf4a75ea-2924-43ca-af92-1cf9cd51def8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"
Nov 02 07:45:00.320861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320796    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmxws\" (UniqueName: \"kubernetes.io/projected/b1e9db58-0abc-488f-8002-49ef49d20cac-kube-api-access-rmxws\") pod \"maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t\" (UID: \"b1e9db58-0abc-488f-8002-49ef49d20cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"
Nov 02 07:45:00.320861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320824    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psxdj\" (UniqueName: \"kubernetes.io/projected/bf4a75ea-2924-43ca-af92-1cf9cd51def8-kube-api-access-psxdj\") pod \"maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9\" (UID: \"bf4a75ea-2924-43ca-af92-1cf9cd51def8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"
Nov 02 07:45:00.321168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.321144    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82934468-ad39-4323-91c5-37163776131f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4\" (UID: \"82934468-ad39-4323-91c5-37163776131f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"
Nov 02 07:45:00.321246 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.320879    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e9db58-0abc-488f-8002-49ef49d20cac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t\" (UID: \"b1e9db58-0abc-488f-8002-49ef49d20cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"
Nov 02 07:45:00.321246 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.321219    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs8bx\" (UniqueName: \"kubernetes.io/projected/234371f3-b3bf-4203-9dd3-d9dc57e666d0-kube-api-access-gs8bx\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf\" (UID: \"234371f3-b3bf-4203-9dd3-d9dc57e666d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"
Nov 02 07:45:00.321331 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.321283    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824\" (UID: \"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"
Nov 02 07:45:00.321370 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.321357    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc34509-c942-498b-9324-ae77e0b22c03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7\" (UID: \"fdc34509-c942-498b-9324-ae77e0b22c03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"
Nov 02 07:45:00.321521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.321500    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4a75ea-2924-43ca-af92-1cf9cd51def8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9\" (UID: \"bf4a75ea-2924-43ca-af92-1cf9cd51def8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"
Nov 02 07:45:00.321734 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.321719    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e9db58-0abc-488f-8002-49ef49d20cac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t\" (UID: \"b1e9db58-0abc-488f-8002-49ef49d20cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"
Nov 02 07:45:00.344530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.344498    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glm9g\" (UniqueName: \"kubernetes.io/projected/fdc34509-c942-498b-9324-ae77e0b22c03-kube-api-access-glm9g\") pod \"maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7\" (UID: \"fdc34509-c942-498b-9324-ae77e0b22c03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"
Nov 02 07:45:00.344658 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.344566    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psxdj\" (UniqueName: \"kubernetes.io/projected/bf4a75ea-2924-43ca-af92-1cf9cd51def8-kube-api-access-psxdj\") pod \"maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9\" (UID: \"bf4a75ea-2924-43ca-af92-1cf9cd51def8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"
Nov 02 07:45:00.344728 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.344711    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv95h\" (UniqueName: \"kubernetes.io/projected/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-kube-api-access-jv95h\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824\" (UID: \"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"
Nov 02 07:45:00.345141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.345119    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxcg\" (UniqueName: \"kubernetes.io/projected/82934468-ad39-4323-91c5-37163776131f-kube-api-access-xhxcg\") pod \"maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4\" (UID: \"82934468-ad39-4323-91c5-37163776131f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"
Nov 02 07:45:00.345750 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.345730    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmxws\" (UniqueName: \"kubernetes.io/projected/b1e9db58-0abc-488f-8002-49ef49d20cac-kube-api-access-rmxws\") pod \"maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t\" (UID: \"b1e9db58-0abc-488f-8002-49ef49d20cac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"
Nov 02 07:45:00.422274 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.422186    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjth\" (UniqueName: \"kubernetes.io/projected/626b6a54-25a8-4149-86f8-b2495d96b931-kube-api-access-nzjth\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp\" (UID: \"626b6a54-25a8-4149-86f8-b2495d96b931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"
Nov 02 07:45:00.422274 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.422264    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8bx\" (UniqueName: \"kubernetes.io/projected/234371f3-b3bf-4203-9dd3-d9dc57e666d0-kube-api-access-gs8bx\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf\" (UID: \"234371f3-b3bf-4203-9dd3-d9dc57e666d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"
Nov 02 07:45:00.422493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.422285    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234371f3-b3bf-4203-9dd3-d9dc57e666d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf\" (UID: \"234371f3-b3bf-4203-9dd3-d9dc57e666d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"
Nov 02 07:45:00.422493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.422324    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35036d5-6707-415a-ae82-ccd088c58cc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d\" (UID: \"b35036d5-6707-415a-ae82-ccd088c58cc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"
Nov 02 07:45:00.422493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.422384    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c22c\" (UniqueName: \"kubernetes.io/projected/b35036d5-6707-415a-ae82-ccd088c58cc7-kube-api-access-4c22c\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d\" (UID: \"b35036d5-6707-415a-ae82-ccd088c58cc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"
Nov 02 07:45:00.422631 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.422533    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/626b6a54-25a8-4149-86f8-b2495d96b931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp\" (UID: \"626b6a54-25a8-4149-86f8-b2495d96b931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"
Nov 02 07:45:00.422993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.422969    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234371f3-b3bf-4203-9dd3-d9dc57e666d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf\" (UID: \"234371f3-b3bf-4203-9dd3-d9dc57e666d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"
Nov 02 07:45:00.423125 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.423102    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35036d5-6707-415a-ae82-ccd088c58cc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d\" (UID: \"b35036d5-6707-415a-ae82-ccd088c58cc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"
Nov 02 07:45:00.423222 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.423204    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/626b6a54-25a8-4149-86f8-b2495d96b931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp\" (UID: \"626b6a54-25a8-4149-86f8-b2495d96b931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"
Nov 02 07:45:00.445627 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.445599    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c22c\" (UniqueName: \"kubernetes.io/projected/b35036d5-6707-415a-ae82-ccd088c58cc7-kube-api-access-4c22c\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d\" (UID: \"b35036d5-6707-415a-ae82-ccd088c58cc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"
Nov 02 07:45:00.445627 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.445612    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjth\" (UniqueName: \"kubernetes.io/projected/626b6a54-25a8-4149-86f8-b2495d96b931-kube-api-access-nzjth\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp\" (UID: \"626b6a54-25a8-4149-86f8-b2495d96b931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"
Nov 02 07:45:00.445784 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.445717    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs8bx\" (UniqueName: \"kubernetes.io/projected/234371f3-b3bf-4203-9dd3-d9dc57e666d0-kube-api-access-gs8bx\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf\" (UID: \"234371f3-b3bf-4203-9dd3-d9dc57e666d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"
Nov 02 07:45:00.488044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.488018    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"
Nov 02 07:45:00.500358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.500329    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"
Nov 02 07:45:00.548503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.548472    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"
Nov 02 07:45:00.554175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.554118    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"
Nov 02 07:45:00.566522 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.566328    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"
Nov 02 07:45:00.580442 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.578101    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"
Nov 02 07:45:00.586158 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.586132    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"
Nov 02 07:45:00.598852 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.598832    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"
Nov 02 07:45:00.827570 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.827532    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"]
Nov 02 07:45:00.922834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:00.922800    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824" event={"ID":"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c","Type":"ContainerStarted","Data":"51ce3e39b42f90f688df15a29265c1ce00b1b45884aa4c57c837576f12a7f508"}
Nov 02 07:45:01.056142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.056111    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"]
Nov 02 07:45:01.060220 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:45:01.060158    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82934468_ad39_4323_91c5_37163776131f.slice/crio-8570ca866350bd45d4109ae712ea4e1557b0111e874352f5d31a774c65b27904 WatchSource:0}: Error finding container 8570ca866350bd45d4109ae712ea4e1557b0111e874352f5d31a774c65b27904: Status 404 returned error can't find the container with id 8570ca866350bd45d4109ae712ea4e1557b0111e874352f5d31a774c65b27904
Nov 02 07:45:01.105129 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:45:01.105074    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626b6a54_25a8_4149_86f8_b2495d96b931.slice/crio-083b4c6c11de422ce5192b96c586d145fde4c217c7b0a5ef6ceb33f7a247204a WatchSource:0}: Error finding container 083b4c6c11de422ce5192b96c586d145fde4c217c7b0a5ef6ceb33f7a247204a: Status 404 returned error can't find the container with id 083b4c6c11de422ce5192b96c586d145fde4c217c7b0a5ef6ceb33f7a247204a
Nov 02 07:45:01.105436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.105343    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"]
Nov 02 07:45:01.115020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.114969    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"]
Nov 02 07:45:01.122841 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.122818    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"]
Nov 02 07:45:01.124113 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:45:01.124088    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234371f3_b3bf_4203_9dd3_d9dc57e666d0.slice/crio-d33f30c17a2751080e6b8fd55457ff15efba03007a1ad6bd2576b2494402ec0f WatchSource:0}: Error finding container d33f30c17a2751080e6b8fd55457ff15efba03007a1ad6bd2576b2494402ec0f: Status 404 returned error can't find the container with id d33f30c17a2751080e6b8fd55457ff15efba03007a1ad6bd2576b2494402ec0f
Nov 02 07:45:01.149489 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:45:01.149458    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4a75ea_2924_43ca_af92_1cf9cd51def8.slice/crio-c6c853e018c3b118cdd25939a2f32167ae62458ab3e3a4e527aeaaedc5833555 WatchSource:0}: Error finding container c6c853e018c3b118cdd25939a2f32167ae62458ab3e3a4e527aeaaedc5833555: Status 404 returned error can't find the container with id c6c853e018c3b118cdd25939a2f32167ae62458ab3e3a4e527aeaaedc5833555
Nov 02 07:45:01.150075 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.149983    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"]
Nov 02 07:45:01.233412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.233384    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"]
Nov 02 07:45:01.269333 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.269306    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"]
Nov 02 07:45:01.933347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.933309    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d" event={"ID":"b35036d5-6707-415a-ae82-ccd088c58cc7","Type":"ContainerStarted","Data":"b1c4391a564107bc44b8167d25b5c0f0b56fd5b86af913a3e89dedc7e7b52a94"}
Nov 02 07:45:01.933513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.933376    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d" event={"ID":"b35036d5-6707-415a-ae82-ccd088c58cc7","Type":"ContainerStarted","Data":"cfc5b74a6e85b31513a67a876a138d2495b312e423a3c2b98d636302103ea592"}
Nov 02 07:45:01.935233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.935206    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824" event={"ID":"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c","Type":"ContainerStarted","Data":"4c1fe26fa8dbe194152866ffb659105a3d01e31b7051115f9aafde63f51ef5f8"}
Nov 02 07:45:01.936579 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.936555    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9" event={"ID":"bf4a75ea-2924-43ca-af92-1cf9cd51def8","Type":"ContainerStarted","Data":"c6c853e018c3b118cdd25939a2f32167ae62458ab3e3a4e527aeaaedc5833555"}
Nov 02 07:45:01.938462 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.938439    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf" event={"ID":"234371f3-b3bf-4203-9dd3-d9dc57e666d0","Type":"ContainerStarted","Data":"e369369b551018805d70224960cf82dbd819f276e1d94d89b768fcd21e301dc1"}
Nov 02 07:45:01.938548 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.938482    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf" event={"ID":"234371f3-b3bf-4203-9dd3-d9dc57e666d0","Type":"ContainerStarted","Data":"d33f30c17a2751080e6b8fd55457ff15efba03007a1ad6bd2576b2494402ec0f"}
Nov 02 07:45:01.939821 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.939796    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7" event={"ID":"fdc34509-c942-498b-9324-ae77e0b22c03","Type":"ContainerStarted","Data":"97b64f48417006afbba3893c41f665b9781f59e501ad1efb2cfed39c040f33f0"}
Nov 02 07:45:01.941472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.941451    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t" event={"ID":"b1e9db58-0abc-488f-8002-49ef49d20cac","Type":"ContainerStarted","Data":"169d243791bec9967125e4b5f6e4bcf896c20fd210324cbad9b2f81efacaf19c"}
Nov 02 07:45:01.941574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.941496    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t" event={"ID":"b1e9db58-0abc-488f-8002-49ef49d20cac","Type":"ContainerStarted","Data":"72609ff7ac31f0acd1c0365b730437b08ffe3c11bd487d1be79d1b3fce8ce035"}
Nov 02 07:45:01.943080 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.943060    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp" event={"ID":"626b6a54-25a8-4149-86f8-b2495d96b931","Type":"ContainerStarted","Data":"f04188e63dca5ecba4a2a5661aa99c4bb16a6818dd104063a402f961e2a8a9a1"}
Nov 02 07:45:01.943178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.943094    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp" event={"ID":"626b6a54-25a8-4149-86f8-b2495d96b931","Type":"ContainerStarted","Data":"083b4c6c11de422ce5192b96c586d145fde4c217c7b0a5ef6ceb33f7a247204a"}
Nov 02 07:45:01.944643 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.944620    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4" event={"ID":"82934468-ad39-4323-91c5-37163776131f","Type":"ContainerStarted","Data":"13ffac8393e0781b35c4f1510ee5c56bea1fa395c1cc585c57c0814b263d3e61"}
Nov 02 07:45:01.944719 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.944657    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4" event={"ID":"82934468-ad39-4323-91c5-37163776131f","Type":"ContainerStarted","Data":"8570ca866350bd45d4109ae712ea4e1557b0111e874352f5d31a774c65b27904"}
Nov 02 07:45:01.986820 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:01.986767    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d" podStartSLOduration=1.986749404 podStartE2EDuration="1.986749404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:01.978466506 +0000 UTC m=+1252.668793468" watchObservedRunningTime="2026-11-02 07:45:01.986749404 +0000 UTC m=+1252.677076369"
Nov 02 07:45:02.037533 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.037415    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824" podStartSLOduration=2.037396372 podStartE2EDuration="2.037396372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:02.03281893 +0000 UTC m=+1252.723145897" watchObservedRunningTime="2026-11-02 07:45:02.037396372 +0000 UTC m=+1252.727723340"
Nov 02 07:45:02.118096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.118049    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t" podStartSLOduration=2.118034292 podStartE2EDuration="2.118034292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:02.076101594 +0000 UTC m=+1252.766428558" watchObservedRunningTime="2026-11-02 07:45:02.118034292 +0000 UTC m=+1252.808361255"
Nov 02 07:45:02.171933 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.171850    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp" podStartSLOduration=2.1718306370000002 podStartE2EDuration="2.171830637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:02.116450515 +0000 UTC m=+1252.806777480" watchObservedRunningTime="2026-11-02 07:45:02.171830637 +0000 UTC m=+1252.862157601"
Nov 02 07:45:02.213212 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.213156    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4" podStartSLOduration=2.213141943 podStartE2EDuration="2.213141943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:02.168552875 +0000 UTC m=+1252.858879843" watchObservedRunningTime="2026-11-02 07:45:02.213141943 +0000 UTC m=+1252.903468907"
Nov 02 07:45:02.213415 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.213239    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf" podStartSLOduration=2.213235629 podStartE2EDuration="2.213235629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:02.208420193 +0000 UTC m=+1252.898747182" watchObservedRunningTime="2026-11-02 07:45:02.213235629 +0000 UTC m=+1252.903562615"
Nov 02 07:45:02.959285 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.959245    4898 generic.go:356] "Generic (PLEG): container finished" podID="626b6a54-25a8-4149-86f8-b2495d96b931" containerID="f04188e63dca5ecba4a2a5661aa99c4bb16a6818dd104063a402f961e2a8a9a1" exitCode=60
Nov 02 07:45:02.959746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.959331    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp" event={"ID":"626b6a54-25a8-4149-86f8-b2495d96b931","Type":"ContainerDied","Data":"f04188e63dca5ecba4a2a5661aa99c4bb16a6818dd104063a402f961e2a8a9a1"}
Nov 02 07:45:02.959943 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.959925    4898 scope.go:122] "RemoveContainer" containerID="f04188e63dca5ecba4a2a5661aa99c4bb16a6818dd104063a402f961e2a8a9a1"
Nov 02 07:45:02.962086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.962063    4898 generic.go:356] "Generic (PLEG): container finished" podID="82934468-ad39-4323-91c5-37163776131f" containerID="13ffac8393e0781b35c4f1510ee5c56bea1fa395c1cc585c57c0814b263d3e61" exitCode=60
Nov 02 07:45:02.962206 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.962178    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4" event={"ID":"82934468-ad39-4323-91c5-37163776131f","Type":"ContainerDied","Data":"13ffac8393e0781b35c4f1510ee5c56bea1fa395c1cc585c57c0814b263d3e61"}
Nov 02 07:45:02.962511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.962481    4898 scope.go:122] "RemoveContainer" containerID="13ffac8393e0781b35c4f1510ee5c56bea1fa395c1cc585c57c0814b263d3e61"
Nov 02 07:45:02.964650 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.964448    4898 generic.go:356] "Generic (PLEG): container finished" podID="b35036d5-6707-415a-ae82-ccd088c58cc7" containerID="b1c4391a564107bc44b8167d25b5c0f0b56fd5b86af913a3e89dedc7e7b52a94" exitCode=60
Nov 02 07:45:02.964650 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.964590    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d" event={"ID":"b35036d5-6707-415a-ae82-ccd088c58cc7","Type":"ContainerDied","Data":"b1c4391a564107bc44b8167d25b5c0f0b56fd5b86af913a3e89dedc7e7b52a94"}
Nov 02 07:45:02.964935 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.964917    4898 scope.go:122] "RemoveContainer" containerID="b1c4391a564107bc44b8167d25b5c0f0b56fd5b86af913a3e89dedc7e7b52a94"
Nov 02 07:45:02.967829 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.967777    4898 generic.go:356] "Generic (PLEG): container finished" podID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" containerID="4c1fe26fa8dbe194152866ffb659105a3d01e31b7051115f9aafde63f51ef5f8" exitCode=60
Nov 02 07:45:02.968008 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.967857    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824" event={"ID":"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c","Type":"ContainerDied","Data":"4c1fe26fa8dbe194152866ffb659105a3d01e31b7051115f9aafde63f51ef5f8"}
Nov 02 07:45:02.968233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.968201    4898 scope.go:122] "RemoveContainer" containerID="4c1fe26fa8dbe194152866ffb659105a3d01e31b7051115f9aafde63f51ef5f8"
Nov 02 07:45:02.971381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.971357    4898 generic.go:356] "Generic (PLEG): container finished" podID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" containerID="62ccb671ec95276ae065f460bf200876471a429421671ebd37c662c3de80f0d5" exitCode=60
Nov 02 07:45:02.971614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.971597    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9" event={"ID":"bf4a75ea-2924-43ca-af92-1cf9cd51def8","Type":"ContainerDied","Data":"62ccb671ec95276ae065f460bf200876471a429421671ebd37c662c3de80f0d5"}
Nov 02 07:45:02.971964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.971944    4898 scope.go:122] "RemoveContainer" containerID="62ccb671ec95276ae065f460bf200876471a429421671ebd37c662c3de80f0d5"
Nov 02 07:45:02.974234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.974210    4898 generic.go:356] "Generic (PLEG): container finished" podID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" containerID="e369369b551018805d70224960cf82dbd819f276e1d94d89b768fcd21e301dc1" exitCode=60
Nov 02 07:45:02.974327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.974289    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf" event={"ID":"234371f3-b3bf-4203-9dd3-d9dc57e666d0","Type":"ContainerDied","Data":"e369369b551018805d70224960cf82dbd819f276e1d94d89b768fcd21e301dc1"}
Nov 02 07:45:02.974607 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.974590    4898 scope.go:122] "RemoveContainer" containerID="e369369b551018805d70224960cf82dbd819f276e1d94d89b768fcd21e301dc1"
Nov 02 07:45:02.977567 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.977545    4898 generic.go:356] "Generic (PLEG): container finished" podID="fdc34509-c942-498b-9324-ae77e0b22c03" containerID="4c5dcdb7d8adbe4f05c492926823fd6ffa3962e4e49fe423f810b5155b3876c2" exitCode=60
Nov 02 07:45:02.977668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.977643    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7" event={"ID":"fdc34509-c942-498b-9324-ae77e0b22c03","Type":"ContainerDied","Data":"4c5dcdb7d8adbe4f05c492926823fd6ffa3962e4e49fe423f810b5155b3876c2"}
Nov 02 07:45:02.977860 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.977842    4898 scope.go:122] "RemoveContainer" containerID="4c5dcdb7d8adbe4f05c492926823fd6ffa3962e4e49fe423f810b5155b3876c2"
Nov 02 07:45:02.997929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.994147    4898 generic.go:356] "Generic (PLEG): container finished" podID="b1e9db58-0abc-488f-8002-49ef49d20cac" containerID="169d243791bec9967125e4b5f6e4bcf896c20fd210324cbad9b2f81efacaf19c" exitCode=60
Nov 02 07:45:02.997929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.994186    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t" event={"ID":"b1e9db58-0abc-488f-8002-49ef49d20cac","Type":"ContainerDied","Data":"169d243791bec9967125e4b5f6e4bcf896c20fd210324cbad9b2f81efacaf19c"}
Nov 02 07:45:02.997929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:02.994791    4898 scope.go:122] "RemoveContainer" containerID="169d243791bec9967125e4b5f6e4bcf896c20fd210324cbad9b2f81efacaf19c"
Nov 02 07:45:03.231248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:03.231169    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"]
Nov 02 07:45:03.279665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:03.279261    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"]
Nov 02 07:45:04.051959 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.034171    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf" event={"ID":"234371f3-b3bf-4203-9dd3-d9dc57e666d0","Type":"ContainerStarted","Data":"f034610f161a9556217a8c7994355c1f073a0c2d71259ed565404522c947f71e"}
Nov 02 07:45:04.051959 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.038421    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7" event={"ID":"fdc34509-c942-498b-9324-ae77e0b22c03","Type":"ContainerStarted","Data":"955b21f48ce054d49ffe01fb1a295d76089cb40d8220a76a8e65ecd20c80d7ba"}
Nov 02 07:45:04.083911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.081906    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t" event={"ID":"b1e9db58-0abc-488f-8002-49ef49d20cac","Type":"ContainerStarted","Data":"103117473f64cc022ce27867a4a8741f6d7b42e10f91303dae56767b9cafcb41"}
Nov 02 07:45:04.105363 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.100513    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp" event={"ID":"626b6a54-25a8-4149-86f8-b2495d96b931","Type":"ContainerStarted","Data":"23ecc8fb9172718b96b06a39ab0d56a46c9c9f8d45ae9b0fc4af8e5c4ad95f35"}
Nov 02 07:45:04.140020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.137884    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4" event={"ID":"82934468-ad39-4323-91c5-37163776131f","Type":"ContainerStarted","Data":"2019a89ba4404cab498c2bdcbcecd59ac74306877daf2314a3dba21b5b3aafa8"}
Nov 02 07:45:04.146192 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.141671    4898 generic.go:356] "Generic (PLEG): container finished" podID="b35036d5-6707-415a-ae82-ccd088c58cc7" containerID="089737e302aae4ed6ae688e80eec90041d1c93e00fcb21a01c33ddddb99f66fb" exitCode=60
Nov 02 07:45:04.146192 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.141814    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d" event={"ID":"b35036d5-6707-415a-ae82-ccd088c58cc7","Type":"ContainerDied","Data":"089737e302aae4ed6ae688e80eec90041d1c93e00fcb21a01c33ddddb99f66fb"}
Nov 02 07:45:04.146192 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.141865    4898 scope.go:122] "RemoveContainer" containerID="b1c4391a564107bc44b8167d25b5c0f0b56fd5b86af913a3e89dedc7e7b52a94"
Nov 02 07:45:04.146192 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.142378    4898 scope.go:122] "RemoveContainer" containerID="089737e302aae4ed6ae688e80eec90041d1c93e00fcb21a01c33ddddb99f66fb"
Nov 02 07:45:04.150472 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:04.150080    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d_odh-ai-gateway-infra(b35036d5-6707-415a-ae82-ccd088c58cc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7"
Nov 02 07:45:04.210261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.209370    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824" event={"ID":"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c","Type":"ContainerStarted","Data":"abe372573149d29b6526a31ed0f3ad475cb1526f745e4dd3d9858801f850ea9d"}
Nov 02 07:45:04.214633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.214536    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9" event={"ID":"bf4a75ea-2924-43ca-af92-1cf9cd51def8","Type":"ContainerStarted","Data":"2f5f6412384394bf0286b114da032d4cbe8e50ce2f248c8401703f4b88e5c300"}
Nov 02 07:45:04.291560 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.291497    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7" podStartSLOduration=4.291483084 podStartE2EDuration="4.291483084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:04.288074314 +0000 UTC m=+1254.978401278" watchObservedRunningTime="2026-11-02 07:45:04.291483084 +0000 UTC m=+1254.981810092"
Nov 02 07:45:04.491655 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:04.491574    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9" podStartSLOduration=4.491560011 podStartE2EDuration="4.491560011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:04.490621944 +0000 UTC m=+1255.180948907" watchObservedRunningTime="2026-11-02 07:45:04.491560011 +0000 UTC m=+1255.181886974"
Nov 02 07:45:05.232505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.232463    4898 generic.go:356] "Generic (PLEG): container finished" podID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" containerID="f034610f161a9556217a8c7994355c1f073a0c2d71259ed565404522c947f71e" exitCode=60
Nov 02 07:45:05.233041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.232549    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf" event={"ID":"234371f3-b3bf-4203-9dd3-d9dc57e666d0","Type":"ContainerDied","Data":"f034610f161a9556217a8c7994355c1f073a0c2d71259ed565404522c947f71e"}
Nov 02 07:45:05.233041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.232633    4898 scope.go:122] "RemoveContainer" containerID="e369369b551018805d70224960cf82dbd819f276e1d94d89b768fcd21e301dc1"
Nov 02 07:45:05.233135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.233073    4898 scope.go:122] "RemoveContainer" containerID="f034610f161a9556217a8c7994355c1f073a0c2d71259ed565404522c947f71e"
Nov 02 07:45:05.233504 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:05.233476    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf_odh-ai-gateway-infra(234371f3-b3bf-4203-9dd3-d9dc57e666d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0"
Nov 02 07:45:05.236039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.235986    4898 generic.go:356] "Generic (PLEG): container finished" podID="fdc34509-c942-498b-9324-ae77e0b22c03" containerID="955b21f48ce054d49ffe01fb1a295d76089cb40d8220a76a8e65ecd20c80d7ba" exitCode=60
Nov 02 07:45:05.236039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.236021    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7" event={"ID":"fdc34509-c942-498b-9324-ae77e0b22c03","Type":"ContainerDied","Data":"955b21f48ce054d49ffe01fb1a295d76089cb40d8220a76a8e65ecd20c80d7ba"}
Nov 02 07:45:05.236660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.236639    4898 scope.go:122] "RemoveContainer" containerID="955b21f48ce054d49ffe01fb1a295d76089cb40d8220a76a8e65ecd20c80d7ba"
Nov 02 07:45:05.237286 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:05.237261    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7_odh-ai-gateway-infra(fdc34509-c942-498b-9324-ae77e0b22c03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7" podUID="fdc34509-c942-498b-9324-ae77e0b22c03"
Nov 02 07:45:05.238686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.238665    4898 generic.go:356] "Generic (PLEG): container finished" podID="b1e9db58-0abc-488f-8002-49ef49d20cac" containerID="103117473f64cc022ce27867a4a8741f6d7b42e10f91303dae56767b9cafcb41" exitCode=60
Nov 02 07:45:05.238778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.238710    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t" event={"ID":"b1e9db58-0abc-488f-8002-49ef49d20cac","Type":"ContainerDied","Data":"103117473f64cc022ce27867a4a8741f6d7b42e10f91303dae56767b9cafcb41"}
Nov 02 07:45:05.239073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.239054    4898 scope.go:122] "RemoveContainer" containerID="103117473f64cc022ce27867a4a8741f6d7b42e10f91303dae56767b9cafcb41"
Nov 02 07:45:05.239883 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:05.239858    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t_odh-ai-gateway-infra(b1e9db58-0abc-488f-8002-49ef49d20cac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac"
Nov 02 07:45:05.241452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.241429    4898 generic.go:356] "Generic (PLEG): container finished" podID="626b6a54-25a8-4149-86f8-b2495d96b931" containerID="23ecc8fb9172718b96b06a39ab0d56a46c9c9f8d45ae9b0fc4af8e5c4ad95f35" exitCode=60
Nov 02 07:45:05.241551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.241493    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp" event={"ID":"626b6a54-25a8-4149-86f8-b2495d96b931","Type":"ContainerDied","Data":"23ecc8fb9172718b96b06a39ab0d56a46c9c9f8d45ae9b0fc4af8e5c4ad95f35"}
Nov 02 07:45:05.241878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.241861    4898 scope.go:122] "RemoveContainer" containerID="23ecc8fb9172718b96b06a39ab0d56a46c9c9f8d45ae9b0fc4af8e5c4ad95f35"
Nov 02 07:45:05.242263 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:05.242237    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp_odh-ai-gateway-infra(626b6a54-25a8-4149-86f8-b2495d96b931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp" podUID="626b6a54-25a8-4149-86f8-b2495d96b931"
Nov 02 07:45:05.243653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.243629    4898 generic.go:356] "Generic (PLEG): container finished" podID="82934468-ad39-4323-91c5-37163776131f" containerID="2019a89ba4404cab498c2bdcbcecd59ac74306877daf2314a3dba21b5b3aafa8" exitCode=60
Nov 02 07:45:05.243780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.243754    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4" event={"ID":"82934468-ad39-4323-91c5-37163776131f","Type":"ContainerDied","Data":"2019a89ba4404cab498c2bdcbcecd59ac74306877daf2314a3dba21b5b3aafa8"}
Nov 02 07:45:05.243976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.243957    4898 scope.go:122] "RemoveContainer" containerID="2019a89ba4404cab498c2bdcbcecd59ac74306877daf2314a3dba21b5b3aafa8"
Nov 02 07:45:05.244276 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:05.244250    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4_odh-ai-gateway-infra(82934468-ad39-4323-91c5-37163776131f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4" podUID="82934468-ad39-4323-91c5-37163776131f"
Nov 02 07:45:05.246235 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.246214    4898 scope.go:122] "RemoveContainer" containerID="089737e302aae4ed6ae688e80eec90041d1c93e00fcb21a01c33ddddb99f66fb"
Nov 02 07:45:05.246531 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:05.246506    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d_odh-ai-gateway-infra(b35036d5-6707-415a-ae82-ccd088c58cc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7"
Nov 02 07:45:05.248749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.248726    4898 generic.go:356] "Generic (PLEG): container finished" podID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" containerID="abe372573149d29b6526a31ed0f3ad475cb1526f745e4dd3d9858801f850ea9d" exitCode=60
Nov 02 07:45:05.248851 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.248816    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824" event={"ID":"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c","Type":"ContainerDied","Data":"abe372573149d29b6526a31ed0f3ad475cb1526f745e4dd3d9858801f850ea9d"}
Nov 02 07:45:05.249170 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.249152    4898 scope.go:122] "RemoveContainer" containerID="abe372573149d29b6526a31ed0f3ad475cb1526f745e4dd3d9858801f850ea9d"
Nov 02 07:45:05.249451 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:05.249430    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824_odh-ai-gateway-infra(1d755c19-a945-4f3b-97c5-f2b6bac1ee8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c"
Nov 02 07:45:05.251339 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.251316    4898 generic.go:356] "Generic (PLEG): container finished" podID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" containerID="2f5f6412384394bf0286b114da032d4cbe8e50ce2f248c8401703f4b88e5c300" exitCode=60
Nov 02 07:45:05.251436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.251389    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9" event={"ID":"bf4a75ea-2924-43ca-af92-1cf9cd51def8","Type":"ContainerDied","Data":"2f5f6412384394bf0286b114da032d4cbe8e50ce2f248c8401703f4b88e5c300"}
Nov 02 07:45:05.251747 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.251626    4898 scope.go:122] "RemoveContainer" containerID="2f5f6412384394bf0286b114da032d4cbe8e50ce2f248c8401703f4b88e5c300"
Nov 02 07:45:05.252004 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.251984    4898 scope.go:122] "RemoveContainer" containerID="4c5dcdb7d8adbe4f05c492926823fd6ffa3962e4e49fe423f810b5155b3876c2"
Nov 02 07:45:05.252085 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:05.251982    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9_odh-ai-gateway-infra(bf4a75ea-2924-43ca-af92-1cf9cd51def8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8"
Nov 02 07:45:05.268364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.268338    4898 scope.go:122] "RemoveContainer" containerID="169d243791bec9967125e4b5f6e4bcf896c20fd210324cbad9b2f81efacaf19c"
Nov 02 07:45:05.293579 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.293547    4898 scope.go:122] "RemoveContainer" containerID="f04188e63dca5ecba4a2a5661aa99c4bb16a6818dd104063a402f961e2a8a9a1"
Nov 02 07:45:05.310619 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.310593    4898 scope.go:122] "RemoveContainer" containerID="13ffac8393e0781b35c4f1510ee5c56bea1fa395c1cc585c57c0814b263d3e61"
Nov 02 07:45:05.363842 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.360465    4898 scope.go:122] "RemoveContainer" containerID="4c1fe26fa8dbe194152866ffb659105a3d01e31b7051115f9aafde63f51ef5f8"
Nov 02 07:45:05.389373 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.389024    4898 scope.go:122] "RemoveContainer" containerID="62ccb671ec95276ae065f460bf200876471a429421671ebd37c662c3de80f0d5"
Nov 02 07:45:05.609979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.609854    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"]
Nov 02 07:45:05.617439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.617412    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"
Nov 02 07:45:05.634756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.634658    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-523de0-dockercfg-57t9r\""
Nov 02 07:45:05.634756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.634719    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-523de0\""
Nov 02 07:45:05.636719 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.635965    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"]
Nov 02 07:45:05.725905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.725830    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f3f205-27d6-4401-be74-e655ca60c83a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl\" (UID: \"07f3f205-27d6-4401-be74-e655ca60c83a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"
Nov 02 07:45:05.726738 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.726710    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdllx\" (UniqueName: \"kubernetes.io/projected/07f3f205-27d6-4401-be74-e655ca60c83a-kube-api-access-xdllx\") pod \"payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl\" (UID: \"07f3f205-27d6-4401-be74-e655ca60c83a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"
Nov 02 07:45:05.764914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.759149    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"]
Nov 02 07:45:05.865018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.833127    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f3f205-27d6-4401-be74-e655ca60c83a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl\" (UID: \"07f3f205-27d6-4401-be74-e655ca60c83a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"
Nov 02 07:45:05.865018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.833286    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdllx\" (UniqueName: \"kubernetes.io/projected/07f3f205-27d6-4401-be74-e655ca60c83a-kube-api-access-xdllx\") pod \"payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl\" (UID: \"07f3f205-27d6-4401-be74-e655ca60c83a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"
Nov 02 07:45:05.865018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.834298    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f3f205-27d6-4401-be74-e655ca60c83a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl\" (UID: \"07f3f205-27d6-4401-be74-e655ca60c83a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"
Nov 02 07:45:05.865018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.851061    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"]
Nov 02 07:45:05.865018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.851219    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"
Nov 02 07:45:05.880926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.868656    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdllx\" (UniqueName: \"kubernetes.io/projected/07f3f205-27d6-4401-be74-e655ca60c83a-kube-api-access-xdllx\") pod \"payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl\" (UID: \"07f3f205-27d6-4401-be74-e655ca60c83a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"
Nov 02 07:45:05.934663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.934629    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be61eaf-5961-4f71-a125-037d47ac2c54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784\" (UID: \"2be61eaf-5961-4f71-a125-037d47ac2c54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"
Nov 02 07:45:05.934831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.934734    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhkb7\" (UniqueName: \"kubernetes.io/projected/2be61eaf-5961-4f71-a125-037d47ac2c54-kube-api-access-jhkb7\") pod \"payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784\" (UID: \"2be61eaf-5961-4f71-a125-037d47ac2c54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"
Nov 02 07:45:05.936079 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:05.936060    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"
Nov 02 07:45:06.041307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.037463    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be61eaf-5961-4f71-a125-037d47ac2c54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784\" (UID: \"2be61eaf-5961-4f71-a125-037d47ac2c54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"
Nov 02 07:45:06.041307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.040091    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhkb7\" (UniqueName: \"kubernetes.io/projected/2be61eaf-5961-4f71-a125-037d47ac2c54-kube-api-access-jhkb7\") pod \"payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784\" (UID: \"2be61eaf-5961-4f71-a125-037d47ac2c54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"
Nov 02 07:45:06.041307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.040813    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be61eaf-5961-4f71-a125-037d47ac2c54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784\" (UID: \"2be61eaf-5961-4f71-a125-037d47ac2c54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"
Nov 02 07:45:06.047497 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.047450    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"]
Nov 02 07:45:06.059634 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.059606    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec9e46e-l44qw"]
Nov 02 07:45:06.073918 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.065993    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhkb7\" (UniqueName: \"kubernetes.io/projected/2be61eaf-5961-4f71-a125-037d47ac2c54-kube-api-access-jhkb7\") pod \"payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784\" (UID: \"2be61eaf-5961-4f71-a125-037d47ac2c54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"
Nov 02 07:45:06.234044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.234012    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"
Nov 02 07:45:06.265498 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.265477    4898 scope.go:122] "RemoveContainer" containerID="f034610f161a9556217a8c7994355c1f073a0c2d71259ed565404522c947f71e"
Nov 02 07:45:06.265949 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:06.265874    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf_odh-ai-gateway-infra(234371f3-b3bf-4203-9dd3-d9dc57e666d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0"
Nov 02 07:45:06.267468 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.267450    4898 scope.go:122] "RemoveContainer" containerID="955b21f48ce054d49ffe01fb1a295d76089cb40d8220a76a8e65ecd20c80d7ba"
Nov 02 07:45:06.267817 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:06.267788    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7_odh-ai-gateway-infra(fdc34509-c942-498b-9324-ae77e0b22c03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7" podUID="fdc34509-c942-498b-9324-ae77e0b22c03"
Nov 02 07:45:06.269635 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.269612    4898 scope.go:122] "RemoveContainer" containerID="103117473f64cc022ce27867a4a8741f6d7b42e10f91303dae56767b9cafcb41"
Nov 02 07:45:06.270009 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:06.269982    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t_odh-ai-gateway-infra(b1e9db58-0abc-488f-8002-49ef49d20cac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac"
Nov 02 07:45:06.272290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.272268    4898 scope.go:122] "RemoveContainer" containerID="23ecc8fb9172718b96b06a39ab0d56a46c9c9f8d45ae9b0fc4af8e5c4ad95f35"
Nov 02 07:45:06.272585 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:06.272561    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp_odh-ai-gateway-infra(626b6a54-25a8-4149-86f8-b2495d96b931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp" podUID="626b6a54-25a8-4149-86f8-b2495d96b931"
Nov 02 07:45:06.274780 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.274759    4898 scope.go:122] "RemoveContainer" containerID="2019a89ba4404cab498c2bdcbcecd59ac74306877daf2314a3dba21b5b3aafa8"
Nov 02 07:45:06.275060 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:06.275038    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4_odh-ai-gateway-infra(82934468-ad39-4323-91c5-37163776131f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4" podUID="82934468-ad39-4323-91c5-37163776131f"
Nov 02 07:45:06.277767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.277738    4898 scope.go:122] "RemoveContainer" containerID="abe372573149d29b6526a31ed0f3ad475cb1526f745e4dd3d9858801f850ea9d"
Nov 02 07:45:06.278087 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:06.278041    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824_odh-ai-gateway-infra(1d755c19-a945-4f3b-97c5-f2b6bac1ee8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c"
Nov 02 07:45:06.280036 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.280018    4898 scope.go:122] "RemoveContainer" containerID="2f5f6412384394bf0286b114da032d4cbe8e50ce2f248c8401703f4b88e5c300"
Nov 02 07:45:06.280298 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:06.280278    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9_odh-ai-gateway-infra(bf4a75ea-2924-43ca-af92-1cf9cd51def8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8"
Nov 02 07:45:06.305990 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:45:06.305872    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f3f205_27d6_4401_be74_e655ca60c83a.slice/crio-0a1c03dba1a0df8adbdb4ba06ac2b3ebc05361c64579deb96b78dd02184b4a3b WatchSource:0}: Error finding container 0a1c03dba1a0df8adbdb4ba06ac2b3ebc05361c64579deb96b78dd02184b4a3b: Status 404 returned error can't find the container with id 0a1c03dba1a0df8adbdb4ba06ac2b3ebc05361c64579deb96b78dd02184b4a3b
Nov 02 07:45:06.348575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.348539    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"]
Nov 02 07:45:06.548480 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:45:06.548446    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be61eaf_5961_4f71_a125_037d47ac2c54.slice/crio-a5df95acdbca984efed7aed99194af32b17d635b1157ad3e90af96b43f83c2ba WatchSource:0}: Error finding container a5df95acdbca984efed7aed99194af32b17d635b1157ad3e90af96b43f83c2ba: Status 404 returned error can't find the container with id a5df95acdbca984efed7aed99194af32b17d635b1157ad3e90af96b43f83c2ba
Nov 02 07:45:06.553321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:06.553297    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"]
Nov 02 07:45:07.296162 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:07.296109    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784" event={"ID":"2be61eaf-5961-4f71-a125-037d47ac2c54","Type":"ContainerStarted","Data":"37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d"}
Nov 02 07:45:07.296663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:07.296213    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784" event={"ID":"2be61eaf-5961-4f71-a125-037d47ac2c54","Type":"ContainerStarted","Data":"a5df95acdbca984efed7aed99194af32b17d635b1157ad3e90af96b43f83c2ba"}
Nov 02 07:45:07.297351 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:07.297327    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"
Nov 02 07:45:07.299587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:07.299552    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl" event={"ID":"07f3f205-27d6-4401-be74-e655ca60c83a","Type":"ContainerStarted","Data":"51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f"}
Nov 02 07:45:07.299707 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:07.299601    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl" event={"ID":"07f3f205-27d6-4401-be74-e655ca60c83a","Type":"ContainerStarted","Data":"0a1c03dba1a0df8adbdb4ba06ac2b3ebc05361c64579deb96b78dd02184b4a3b"}
Nov 02 07:45:07.299823 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:07.299807    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"
Nov 02 07:45:07.339789 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:07.339740    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784" podStartSLOduration=2.339727293 podStartE2EDuration="2.339727293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:07.338369244 +0000 UTC m=+1258.028696207" watchObservedRunningTime="2026-11-02 07:45:07.339727293 +0000 UTC m=+1258.030054253"
Nov 02 07:45:07.471351 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:07.471314    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14c9b913-34dc-4904-a36b-11651b8ef06e" path="/var/lib/kubelet/pods/14c9b913-34dc-4904-a36b-11651b8ef06e/volumes"
Nov 02 07:45:08.233186 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:08.233143    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm" podUID="1663b127-9ab2-45b6-afc5-78bb5236aa47" containerName="payload-processing" containerID="cri-o://5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0" gracePeriod=25
Nov 02 07:45:08.280997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:08.280953    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865" podUID="f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf" containerName="payload-pre-processing" containerID="cri-o://8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f" gracePeriod=25
Nov 02 07:45:09.055956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.045953    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl" podStartSLOduration=4.045933379 podStartE2EDuration="4.045933379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:07.392123403 +0000 UTC m=+1258.082450367" watchObservedRunningTime="2026-11-02 07:45:09.045933379 +0000 UTC m=+1259.736260345"
Nov 02 07:45:09.055956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.050139    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"
Nov 02 07:45:09.070117 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.068640    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"]
Nov 02 07:45:09.073793 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.073765    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d33fa69-vvhsj"]
Nov 02 07:45:09.147178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.147150    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"
Nov 02 07:45:09.150319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.150294    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1663b127-9ab2-45b6-afc5-78bb5236aa47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1663b127-9ab2-45b6-afc5-78bb5236aa47-plugins-config-volume\") pod \"1663b127-9ab2-45b6-afc5-78bb5236aa47\" (UID: \"1663b127-9ab2-45b6-afc5-78bb5236aa47\") "
Nov 02 07:45:09.150466 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.150450    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1663b127-9ab2-45b6-afc5-78bb5236aa47-kube-api-access-vxkkl\" (UniqueName: \"kubernetes.io/projected/1663b127-9ab2-45b6-afc5-78bb5236aa47-kube-api-access-vxkkl\") pod \"1663b127-9ab2-45b6-afc5-78bb5236aa47\" (UID: \"1663b127-9ab2-45b6-afc5-78bb5236aa47\") "
Nov 02 07:45:09.150555 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.150539    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-plugins-config-volume\") pod \"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf\" (UID: \"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf\") "
Nov 02 07:45:09.150659 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.150644    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-kube-api-access-t2sd8\" (UniqueName: \"kubernetes.io/projected/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-kube-api-access-t2sd8\") pod \"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf\" (UID: \"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf\") "
Nov 02 07:45:09.151115 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.150642    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1663b127-9ab2-45b6-afc5-78bb5236aa47-plugins-config-volume" pod "1663b127-9ab2-45b6-afc5-78bb5236aa47" (UID: "1663b127-9ab2-45b6-afc5-78bb5236aa47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:09.151115 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.151019    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-plugins-config-volume" pod "f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf" (UID: "f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:09.151442 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.151423    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1663b127-9ab2-45b6-afc5-78bb5236aa47-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:09.151492 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.151449    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:09.152624 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.152597    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1663b127-9ab2-45b6-afc5-78bb5236aa47-kube-api-access-vxkkl" pod "1663b127-9ab2-45b6-afc5-78bb5236aa47" (UID: "1663b127-9ab2-45b6-afc5-78bb5236aa47"). InnerVolumeSpecName "kube-api-access-vxkkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:09.152736 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.152655    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-kube-api-access-t2sd8" pod "f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf" (UID: "f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf"). InnerVolumeSpecName "kube-api-access-t2sd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:09.253060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.252974    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2sd8\" (UniqueName: \"kubernetes.io/projected/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf-kube-api-access-t2sd8\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:09.253060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.253008    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxkkl\" (UniqueName: \"kubernetes.io/projected/1663b127-9ab2-45b6-afc5-78bb5236aa47-kube-api-access-vxkkl\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:09.451021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.450979    4898 generic.go:356] "Generic (PLEG): container finished" podID="1663b127-9ab2-45b6-afc5-78bb5236aa47" containerID="5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0" exitCode=0
Nov 02 07:45:09.451210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.451076    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"
Nov 02 07:45:09.451210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.451135    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm" event={"ID":"1663b127-9ab2-45b6-afc5-78bb5236aa47","Type":"ContainerDied","Data":"5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0"}
Nov 02 07:45:09.451210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.451188    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm" event={"ID":"1663b127-9ab2-45b6-afc5-78bb5236aa47","Type":"ContainerDied","Data":"65e97183d744fa4cf9cc3ae4a18fc7a3f0e6d98fd253d1ef708992041abe9d7e"}
Nov 02 07:45:09.451402 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.451220    4898 scope.go:122] "RemoveContainer" containerID="5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0"
Nov 02 07:45:09.453417 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.453360    4898 generic.go:356] "Generic (PLEG): container finished" podID="f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf" containerID="8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f" exitCode=0
Nov 02 07:45:09.453579 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.453528    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"
Nov 02 07:45:09.453675 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.453637    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865" event={"ID":"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf","Type":"ContainerDied","Data":"8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f"}
Nov 02 07:45:09.453742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.453673    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865" event={"ID":"f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf","Type":"ContainerDied","Data":"c1d4419264f28859fb8b6b7ea3854ea980e6828b3853ba6ea6febede0623f60b"}
Nov 02 07:45:09.603097 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.598839    4898 scope.go:122] "RemoveContainer" containerID="5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0"
Nov 02 07:45:09.629925 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:09.600287    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0\": container with ID starting with 5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0 not found: ID does not exist" containerID="5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0"
Nov 02 07:45:09.629925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.629107    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0"} err="failed to get container status \"5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0\": rpc error: code = NotFound desc = could not find container \"5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0\": container with ID starting with 5b381a9f25d399b885cb944d7a8d0b27e7001a32a2a498b1fe0da83f6b91aba0 not found: ID does not exist"
Nov 02 07:45:09.629925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.629137    4898 scope.go:122] "RemoveContainer" containerID="8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f"
Nov 02 07:45:09.648610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.648572    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb4ced7-494b-4050-8a5a-8a604bf056c4" path="/var/lib/kubelet/pods/1cb4ced7-494b-4050-8a5a-8a604bf056c4/volumes"
Nov 02 07:45:09.668259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.668228    4898 scope.go:122] "RemoveContainer" containerID="8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f"
Nov 02 07:45:09.670771 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:09.668629    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f\": container with ID starting with 8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f not found: ID does not exist" containerID="8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f"
Nov 02 07:45:09.670771 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.668667    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f"} err="failed to get container status \"8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f\": rpc error: code = NotFound desc = could not find container \"8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f\": container with ID starting with 8b37d0b424be7d6cfaf494e8c3c6c955eb8259ff2107c2fe6d43c958ef94569f not found: ID does not exist"
Nov 02 07:45:09.690820 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.690792    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"]
Nov 02 07:45:09.702699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.702671    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d12c2-c8b5cd885-gg7gm"]
Nov 02 07:45:09.735798 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.734713    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"]
Nov 02 07:45:09.766913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:09.765707    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d12c2-5878b4575-cg865"]
Nov 02 07:45:10.155445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:10.155409    4898 scope.go:122] "RemoveContainer" containerID="3fec609167b9e84faf32ee5d1d19981849ce5b25ef64d85c0665f5973f2f01af"
Nov 02 07:45:10.685933 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:10.684561    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 07:45:11.044641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:11.044550    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"]
Nov 02 07:45:11.066913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:11.062915    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d19a4f8-g2px7"]
Nov 02 07:45:11.495080 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:11.495050    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1663b127-9ab2-45b6-afc5-78bb5236aa47" path="/var/lib/kubelet/pods/1663b127-9ab2-45b6-afc5-78bb5236aa47/volumes"
Nov 02 07:45:11.496169 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:11.496124    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da36ae40-a2c8-432a-bc54-6c1097e97b1a" path="/var/lib/kubelet/pods/da36ae40-a2c8-432a-bc54-6c1097e97b1a/volumes"
Nov 02 07:45:11.496792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:11.496751    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf" path="/var/lib/kubelet/pods/f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf/volumes"
Nov 02 07:45:12.518501 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:12.516796    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 07:45:15.009388 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:15.009174    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:45:15.459934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.459877    4898 scope.go:122] "RemoveContainer" containerID="2019a89ba4404cab498c2bdcbcecd59ac74306877daf2314a3dba21b5b3aafa8"
Nov 02 07:45:15.733235 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.733123    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"]
Nov 02 07:45:15.734234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.734208    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1663b127-9ab2-45b6-afc5-78bb5236aa47" containerName="payload-processing"
Nov 02 07:45:15.734413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.734398    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1663b127-9ab2-45b6-afc5-78bb5236aa47" containerName="payload-processing"
Nov 02 07:45:15.734543 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.734533    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf" containerName="payload-pre-processing"
Nov 02 07:45:15.734615 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.734605    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf" containerName="payload-pre-processing"
Nov 02 07:45:15.734913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.734882    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1663b127-9ab2-45b6-afc5-78bb5236aa47" containerName="payload-processing"
Nov 02 07:45:15.735007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.734999    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3a5ca3d-fa04-4acd-b5b9-f02bfd52dbbf" containerName="payload-pre-processing"
Nov 02 07:45:15.740925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.740880    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"
Nov 02 07:45:15.753246 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.753212    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"]
Nov 02 07:45:15.768396 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.768364    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6cc3f7\""
Nov 02 07:45:15.781846 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.780740    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6cc3f7-dockercfg-wxrpd\""
Nov 02 07:45:15.844006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.835311    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d42fa-2002-4206-8747-7a632e407bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c\" (UID: \"e99d42fa-2002-4206-8747-7a632e407bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"
Nov 02 07:45:15.844006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.835483    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bkxn\" (UniqueName: \"kubernetes.io/projected/e99d42fa-2002-4206-8747-7a632e407bfa-kube-api-access-7bkxn\") pod \"payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c\" (UID: \"e99d42fa-2002-4206-8747-7a632e407bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"
Nov 02 07:45:15.955077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.954241    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d42fa-2002-4206-8747-7a632e407bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c\" (UID: \"e99d42fa-2002-4206-8747-7a632e407bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"
Nov 02 07:45:15.955077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.954348    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bkxn\" (UniqueName: \"kubernetes.io/projected/e99d42fa-2002-4206-8747-7a632e407bfa-kube-api-access-7bkxn\") pod \"payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c\" (UID: \"e99d42fa-2002-4206-8747-7a632e407bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"
Nov 02 07:45:15.955878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.955836    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d42fa-2002-4206-8747-7a632e407bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c\" (UID: \"e99d42fa-2002-4206-8747-7a632e407bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"
Nov 02 07:45:15.982423 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:15.982390    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bkxn\" (UniqueName: \"kubernetes.io/projected/e99d42fa-2002-4206-8747-7a632e407bfa-kube-api-access-7bkxn\") pod \"payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c\" (UID: \"e99d42fa-2002-4206-8747-7a632e407bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"
Nov 02 07:45:16.071168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:16.071081    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"
Nov 02 07:45:16.379208 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:45:16.379162    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99d42fa_2002_4206_8747_7a632e407bfa.slice/crio-67d030b2d3e73dad1d18d1128ccaa78019312e96de836e1a117c4022e59cfe52 WatchSource:0}: Error finding container 67d030b2d3e73dad1d18d1128ccaa78019312e96de836e1a117c4022e59cfe52: Status 404 returned error can't find the container with id 67d030b2d3e73dad1d18d1128ccaa78019312e96de836e1a117c4022e59cfe52
Nov 02 07:45:16.382172 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:16.382140    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"]
Nov 02 07:45:16.460930 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:16.457039    4898 scope.go:122] "RemoveContainer" containerID="23ecc8fb9172718b96b06a39ab0d56a46c9c9f8d45ae9b0fc4af8e5c4ad95f35"
Nov 02 07:45:16.460930 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:16.457817    4898 scope.go:122] "RemoveContainer" containerID="089737e302aae4ed6ae688e80eec90041d1c93e00fcb21a01c33ddddb99f66fb"
Nov 02 07:45:16.613958 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:16.613084    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4" event={"ID":"82934468-ad39-4323-91c5-37163776131f","Type":"ContainerStarted","Data":"266873b92285746c92dddf66580d6a6c3bada870e5665163334ddf24c48c9ac5"}
Nov 02 07:45:16.615341 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:16.615311    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c" event={"ID":"e99d42fa-2002-4206-8747-7a632e407bfa","Type":"ContainerStarted","Data":"cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd"}
Nov 02 07:45:16.615457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:16.615364    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c" event={"ID":"e99d42fa-2002-4206-8747-7a632e407bfa","Type":"ContainerStarted","Data":"67d030b2d3e73dad1d18d1128ccaa78019312e96de836e1a117c4022e59cfe52"}
Nov 02 07:45:16.616164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:16.616116    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"
Nov 02 07:45:16.708511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:16.706227    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c" podStartSLOduration=1.706211504 podStartE2EDuration="1.706211504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:16.705781159 +0000 UTC m=+1267.396108121" watchObservedRunningTime="2026-11-02 07:45:16.706211504 +0000 UTC m=+1267.396538470"
Nov 02 07:45:17.458636 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:17.458603    4898 scope.go:122] "RemoveContainer" containerID="abe372573149d29b6526a31ed0f3ad475cb1526f745e4dd3d9858801f850ea9d"
Nov 02 07:45:17.459126 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:17.458832    4898 scope.go:122] "RemoveContainer" containerID="955b21f48ce054d49ffe01fb1a295d76089cb40d8220a76a8e65ecd20c80d7ba"
Nov 02 07:45:17.631292 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:17.631250    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d" event={"ID":"b35036d5-6707-415a-ae82-ccd088c58cc7","Type":"ContainerStarted","Data":"57e57b6dad94543fb28f701332b0a4b86a40dcf61fcabcd5d697c0468a880aac"}
Nov 02 07:45:17.633942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:17.633913    4898 generic.go:356] "Generic (PLEG): container finished" podID="626b6a54-25a8-4149-86f8-b2495d96b931" containerID="8df61b74680eb296ee71c7f565de0ea80a194ee73c86a3bc2e2ccd8d3e0d601a" exitCode=60
Nov 02 07:45:17.634099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:17.633953    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp" event={"ID":"626b6a54-25a8-4149-86f8-b2495d96b931","Type":"ContainerDied","Data":"8df61b74680eb296ee71c7f565de0ea80a194ee73c86a3bc2e2ccd8d3e0d601a"}
Nov 02 07:45:17.634099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:17.634016    4898 scope.go:122] "RemoveContainer" containerID="23ecc8fb9172718b96b06a39ab0d56a46c9c9f8d45ae9b0fc4af8e5c4ad95f35"
Nov 02 07:45:17.634432 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:17.634414    4898 scope.go:122] "RemoveContainer" containerID="8df61b74680eb296ee71c7f565de0ea80a194ee73c86a3bc2e2ccd8d3e0d601a"
Nov 02 07:45:17.634790 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:17.634769    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp_odh-ai-gateway-infra(626b6a54-25a8-4149-86f8-b2495d96b931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp" podUID="626b6a54-25a8-4149-86f8-b2495d96b931"
Nov 02 07:45:17.636454 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:17.636429    4898 generic.go:356] "Generic (PLEG): container finished" podID="82934468-ad39-4323-91c5-37163776131f" containerID="266873b92285746c92dddf66580d6a6c3bada870e5665163334ddf24c48c9ac5" exitCode=60
Nov 02 07:45:17.636609 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:17.636474    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4" event={"ID":"82934468-ad39-4323-91c5-37163776131f","Type":"ContainerDied","Data":"266873b92285746c92dddf66580d6a6c3bada870e5665163334ddf24c48c9ac5"}
Nov 02 07:45:17.636945 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:17.636927    4898 scope.go:122] "RemoveContainer" containerID="266873b92285746c92dddf66580d6a6c3bada870e5665163334ddf24c48c9ac5"
Nov 02 07:45:17.637296 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:17.637275    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4_odh-ai-gateway-infra(82934468-ad39-4323-91c5-37163776131f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4" podUID="82934468-ad39-4323-91c5-37163776131f"
Nov 02 07:45:17.650056 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:17.650036    4898 scope.go:122] "RemoveContainer" containerID="2019a89ba4404cab498c2bdcbcecd59ac74306877daf2314a3dba21b5b3aafa8"
Nov 02 07:45:17.675914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:17.673884    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"]
Nov 02 07:45:18.433804 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:18.432207    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"
Nov 02 07:45:18.459534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:18.459502    4898 scope.go:122] "RemoveContainer" containerID="2f5f6412384394bf0286b114da032d4cbe8e50ce2f248c8401703f4b88e5c300"
Nov 02 07:45:18.461607 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:18.461585    4898 scope.go:122] "RemoveContainer" containerID="f034610f161a9556217a8c7994355c1f073a0c2d71259ed565404522c947f71e"
Nov 02 07:45:18.461781 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:18.461763    4898 scope.go:122] "RemoveContainer" containerID="103117473f64cc022ce27867a4a8741f6d7b42e10f91303dae56767b9cafcb41"
Nov 02 07:45:18.649249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:18.649213    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824" event={"ID":"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c","Type":"ContainerStarted","Data":"4eb754dc3a8263f5ba8041b5b601ec9b2e169017e04d3a369ea08be47ad524c9"}
Nov 02 07:45:18.651685 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:18.651653    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7" event={"ID":"fdc34509-c942-498b-9324-ae77e0b22c03","Type":"ContainerStarted","Data":"3b683debc3a60089ed96d0d4132335243702b02d19be2c2078a41067ad905e22"}
Nov 02 07:45:18.663953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:18.663048    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d" event={"ID":"b35036d5-6707-415a-ae82-ccd088c58cc7","Type":"ContainerDied","Data":"57e57b6dad94543fb28f701332b0a4b86a40dcf61fcabcd5d697c0468a880aac"}
Nov 02 07:45:18.663953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:18.663019    4898 generic.go:356] "Generic (PLEG): container finished" podID="b35036d5-6707-415a-ae82-ccd088c58cc7" containerID="57e57b6dad94543fb28f701332b0a4b86a40dcf61fcabcd5d697c0468a880aac" exitCode=60
Nov 02 07:45:18.663953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:18.663124    4898 scope.go:122] "RemoveContainer" containerID="089737e302aae4ed6ae688e80eec90041d1c93e00fcb21a01c33ddddb99f66fb"
Nov 02 07:45:18.663953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:18.663522    4898 scope.go:122] "RemoveContainer" containerID="57e57b6dad94543fb28f701332b0a4b86a40dcf61fcabcd5d697c0468a880aac"
Nov 02 07:45:18.663953 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:18.663907    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d_odh-ai-gateway-infra(b35036d5-6707-415a-ae82-ccd088c58cc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7"
Nov 02 07:45:18.687514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:18.687413    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"]
Nov 02 07:45:18.724473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:18.724446    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"]
Nov 02 07:45:19.147769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.147697    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"
Nov 02 07:45:19.203166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.201627    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82934468-ad39-4323-91c5-37163776131f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82934468-ad39-4323-91c5-37163776131f-maas-api-service-ca\") pod \"82934468-ad39-4323-91c5-37163776131f\" (UID: \"82934468-ad39-4323-91c5-37163776131f\") "
Nov 02 07:45:19.203166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.201680    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82934468-ad39-4323-91c5-37163776131f-kube-api-access-xhxcg\" (UniqueName: \"kubernetes.io/projected/82934468-ad39-4323-91c5-37163776131f-kube-api-access-xhxcg\") pod \"82934468-ad39-4323-91c5-37163776131f\" (UID: \"82934468-ad39-4323-91c5-37163776131f\") "
Nov 02 07:45:19.203166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.201997    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82934468-ad39-4323-91c5-37163776131f-maas-api-service-ca" pod "82934468-ad39-4323-91c5-37163776131f" (UID: "82934468-ad39-4323-91c5-37163776131f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:19.203166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.202532    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82934468-ad39-4323-91c5-37163776131f-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:19.206939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.204873    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82934468-ad39-4323-91c5-37163776131f-kube-api-access-xhxcg" pod "82934468-ad39-4323-91c5-37163776131f" (UID: "82934468-ad39-4323-91c5-37163776131f"). InnerVolumeSpecName "kube-api-access-xhxcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:19.220137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.220115    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"
Nov 02 07:45:19.303321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.303228    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/626b6a54-25a8-4149-86f8-b2495d96b931-kube-api-access-nzjth\" (UniqueName: \"kubernetes.io/projected/626b6a54-25a8-4149-86f8-b2495d96b931-kube-api-access-nzjth\") pod \"626b6a54-25a8-4149-86f8-b2495d96b931\" (UID: \"626b6a54-25a8-4149-86f8-b2495d96b931\") "
Nov 02 07:45:19.303549 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.303527    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/626b6a54-25a8-4149-86f8-b2495d96b931-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/626b6a54-25a8-4149-86f8-b2495d96b931-maas-api-service-ca\") pod \"626b6a54-25a8-4149-86f8-b2495d96b931\" (UID: \"626b6a54-25a8-4149-86f8-b2495d96b931\") "
Nov 02 07:45:19.303938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.303879    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626b6a54-25a8-4149-86f8-b2495d96b931-maas-api-service-ca" pod "626b6a54-25a8-4149-86f8-b2495d96b931" (UID: "626b6a54-25a8-4149-86f8-b2495d96b931"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:19.304217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.304179    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/626b6a54-25a8-4149-86f8-b2495d96b931-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:19.304312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.304221    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhxcg\" (UniqueName: \"kubernetes.io/projected/82934468-ad39-4323-91c5-37163776131f-kube-api-access-xhxcg\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:19.306324 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.306295    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626b6a54-25a8-4149-86f8-b2495d96b931-kube-api-access-nzjth" pod "626b6a54-25a8-4149-86f8-b2495d96b931" (UID: "626b6a54-25a8-4149-86f8-b2495d96b931"). InnerVolumeSpecName "kube-api-access-nzjth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:19.406144 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.406105    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzjth\" (UniqueName: \"kubernetes.io/projected/626b6a54-25a8-4149-86f8-b2495d96b931-kube-api-access-nzjth\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:19.471227 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.471194    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"
Nov 02 07:45:19.683546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.683512    4898 generic.go:356] "Generic (PLEG): container finished" podID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" containerID="dfbfa61d7ccadff0c276adaa45fe2b53f60c56f0877bccd90b2a299391279d5b" exitCode=60
Nov 02 07:45:19.683776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.683630    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf" event={"ID":"234371f3-b3bf-4203-9dd3-d9dc57e666d0","Type":"ContainerDied","Data":"dfbfa61d7ccadff0c276adaa45fe2b53f60c56f0877bccd90b2a299391279d5b"}
Nov 02 07:45:19.683956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.683826    4898 scope.go:122] "RemoveContainer" containerID="f034610f161a9556217a8c7994355c1f073a0c2d71259ed565404522c947f71e"
Nov 02 07:45:19.684792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.684762    4898 scope.go:122] "RemoveContainer" containerID="dfbfa61d7ccadff0c276adaa45fe2b53f60c56f0877bccd90b2a299391279d5b"
Nov 02 07:45:19.685247 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:19.685223    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf_odh-ai-gateway-infra(234371f3-b3bf-4203-9dd3-d9dc57e666d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0"
Nov 02 07:45:19.687315 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.687290    4898 generic.go:356] "Generic (PLEG): container finished" podID="fdc34509-c942-498b-9324-ae77e0b22c03" containerID="3b683debc3a60089ed96d0d4132335243702b02d19be2c2078a41067ad905e22" exitCode=60
Nov 02 07:45:19.687414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.687354    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7" event={"ID":"fdc34509-c942-498b-9324-ae77e0b22c03","Type":"ContainerDied","Data":"3b683debc3a60089ed96d0d4132335243702b02d19be2c2078a41067ad905e22"}
Nov 02 07:45:19.687782 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.687758    4898 scope.go:122] "RemoveContainer" containerID="3b683debc3a60089ed96d0d4132335243702b02d19be2c2078a41067ad905e22"
Nov 02 07:45:19.688489 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:19.688165    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7_odh-ai-gateway-infra(fdc34509-c942-498b-9324-ae77e0b22c03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7" podUID="fdc34509-c942-498b-9324-ae77e0b22c03"
Nov 02 07:45:19.690771 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.690745    4898 generic.go:356] "Generic (PLEG): container finished" podID="b1e9db58-0abc-488f-8002-49ef49d20cac" containerID="4c4814f546bac716513665cd76f07889552dc52ab61557e20c9a143ef397df52" exitCode=60
Nov 02 07:45:19.690905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.690841    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t" event={"ID":"b1e9db58-0abc-488f-8002-49ef49d20cac","Type":"ContainerDied","Data":"4c4814f546bac716513665cd76f07889552dc52ab61557e20c9a143ef397df52"}
Nov 02 07:45:19.691249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.691228    4898 scope.go:122] "RemoveContainer" containerID="4c4814f546bac716513665cd76f07889552dc52ab61557e20c9a143ef397df52"
Nov 02 07:45:19.691596 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:19.691569    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t_odh-ai-gateway-infra(b1e9db58-0abc-488f-8002-49ef49d20cac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac"
Nov 02 07:45:19.693288 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.693268    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp" event={"ID":"626b6a54-25a8-4149-86f8-b2495d96b931","Type":"ContainerDied","Data":"083b4c6c11de422ce5192b96c586d145fde4c217c7b0a5ef6ceb33f7a247204a"}
Nov 02 07:45:19.693411 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.693269    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"
Nov 02 07:45:19.694865 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.694849    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"
Nov 02 07:45:19.694999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.694978    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4" event={"ID":"82934468-ad39-4323-91c5-37163776131f","Type":"ContainerDied","Data":"8570ca866350bd45d4109ae712ea4e1557b0111e874352f5d31a774c65b27904"}
Nov 02 07:45:19.702174 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.702150    4898 generic.go:356] "Generic (PLEG): container finished" podID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" containerID="4eb754dc3a8263f5ba8041b5b601ec9b2e169017e04d3a369ea08be47ad524c9" exitCode=60
Nov 02 07:45:19.703073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.702242    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824" event={"ID":"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c","Type":"ContainerDied","Data":"4eb754dc3a8263f5ba8041b5b601ec9b2e169017e04d3a369ea08be47ad524c9"}
Nov 02 07:45:19.703073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.702607    4898 scope.go:122] "RemoveContainer" containerID="4eb754dc3a8263f5ba8041b5b601ec9b2e169017e04d3a369ea08be47ad524c9"
Nov 02 07:45:19.703073 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:19.702928    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824_odh-ai-gateway-infra(1d755c19-a945-4f3b-97c5-f2b6bac1ee8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c"
Nov 02 07:45:19.706429 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.706401    4898 generic.go:356] "Generic (PLEG): container finished" podID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" containerID="4a214232c9d6067e13d9c2f9bd3d4999c456922e4e617bde56da438f80c3f8a5" exitCode=60
Nov 02 07:45:19.706670 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.706649    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9" event={"ID":"bf4a75ea-2924-43ca-af92-1cf9cd51def8","Type":"ContainerDied","Data":"4a214232c9d6067e13d9c2f9bd3d4999c456922e4e617bde56da438f80c3f8a5"}
Nov 02 07:45:19.707243 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.707212    4898 scope.go:122] "RemoveContainer" containerID="4a214232c9d6067e13d9c2f9bd3d4999c456922e4e617bde56da438f80c3f8a5"
Nov 02 07:45:19.707662 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:19.707638    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9_odh-ai-gateway-infra(bf4a75ea-2924-43ca-af92-1cf9cd51def8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8"
Nov 02 07:45:19.709797 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.709767    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"]
Nov 02 07:45:19.843389 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.843359    4898 scope.go:122] "RemoveContainer" containerID="955b21f48ce054d49ffe01fb1a295d76089cb40d8220a76a8e65ecd20c80d7ba"
Nov 02 07:45:19.880080 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.880053    4898 scope.go:122] "RemoveContainer" containerID="103117473f64cc022ce27867a4a8741f6d7b42e10f91303dae56767b9cafcb41"
Nov 02 07:45:19.905398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.904377    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"]
Nov 02 07:45:19.919448 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.919415    4898 scope.go:122] "RemoveContainer" containerID="8df61b74680eb296ee71c7f565de0ea80a194ee73c86a3bc2e2ccd8d3e0d601a"
Nov 02 07:45:19.970393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.970358    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"
Nov 02 07:45:19.970624 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.970603    4898 scope.go:122] "RemoveContainer" containerID="266873b92285746c92dddf66580d6a6c3bada870e5665163334ddf24c48c9ac5"
Nov 02 07:45:19.982018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.981992    4898 scope.go:122] "RemoveContainer" containerID="abe372573149d29b6526a31ed0f3ad475cb1526f745e4dd3d9858801f850ea9d"
Nov 02 07:45:19.999816 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:19.999788    4898 scope.go:122] "RemoveContainer" containerID="2f5f6412384394bf0286b114da032d4cbe8e50ce2f248c8401703f4b88e5c300"
Nov 02 07:45:20.017677 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.017632    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b35036d5-6707-415a-ae82-ccd088c58cc7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35036d5-6707-415a-ae82-ccd088c58cc7-maas-api-service-ca\") pod \"b35036d5-6707-415a-ae82-ccd088c58cc7\" (UID: \"b35036d5-6707-415a-ae82-ccd088c58cc7\") "
Nov 02 07:45:20.017849 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.017827    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35036d5-6707-415a-ae82-ccd088c58cc7-kube-api-access-4c22c\" (UniqueName: \"kubernetes.io/projected/b35036d5-6707-415a-ae82-ccd088c58cc7-kube-api-access-4c22c\") pod \"b35036d5-6707-415a-ae82-ccd088c58cc7\" (UID: \"b35036d5-6707-415a-ae82-ccd088c58cc7\") "
Nov 02 07:45:20.018326 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.018299    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35036d5-6707-415a-ae82-ccd088c58cc7-maas-api-service-ca" pod "b35036d5-6707-415a-ae82-ccd088c58cc7" (UID: "b35036d5-6707-415a-ae82-ccd088c58cc7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:20.018715 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.018679    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b35036d5-6707-415a-ae82-ccd088c58cc7-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:20.020617 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.020594    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35036d5-6707-415a-ae82-ccd088c58cc7-kube-api-access-4c22c" pod "b35036d5-6707-415a-ae82-ccd088c58cc7" (UID: "b35036d5-6707-415a-ae82-ccd088c58cc7"). InnerVolumeSpecName "kube-api-access-4c22c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:20.021041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.021025    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"]
Nov 02 07:45:20.025862 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.025838    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893425-pmmcp"]
Nov 02 07:45:20.120126 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.120077    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c22c\" (UniqueName: \"kubernetes.io/projected/b35036d5-6707-415a-ae82-ccd088c58cc7-kube-api-access-4c22c\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:20.247010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.242834    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"]
Nov 02 07:45:20.250769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.250391    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e07c6-29893425-f6qs4"]
Nov 02 07:45:20.721045 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.721012    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"
Nov 02 07:45:20.721045 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.721039    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d" event={"ID":"b35036d5-6707-415a-ae82-ccd088c58cc7","Type":"ContainerDied","Data":"cfc5b74a6e85b31513a67a876a138d2495b312e423a3c2b98d636302103ea592"}
Nov 02 07:45:20.721655 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.721114    4898 scope.go:122] "RemoveContainer" containerID="57e57b6dad94543fb28f701332b0a4b86a40dcf61fcabcd5d697c0468a880aac"
Nov 02 07:45:20.785977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.785923    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"]
Nov 02 07:45:20.810014 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:20.809964    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35036d5_6707_415a_ae82_ccd088c58cc7.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 07:45:20.818878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.818817    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"]
Nov 02 07:45:20.825236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.825181    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893425-p4p8d"]
Nov 02 07:45:20.991420 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:20.983606    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"]
Nov 02 07:45:21.035407 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.035379    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"]
Nov 02 07:45:21.070926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.065452    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"]
Nov 02 07:45:21.070926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.069050    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"
Nov 02 07:45:21.125594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.125524    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5bd9da-6znh9"]
Nov 02 07:45:21.172602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.172263    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"
Nov 02 07:45:21.196970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.195598    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-maas-api-service-ca\") pod \"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c\" (UID: \"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c\") "
Nov 02 07:45:21.196970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.195648    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-kube-api-access-jv95h\" (UniqueName: \"kubernetes.io/projected/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-kube-api-access-jv95h\") pod \"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c\" (UID: \"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c\") "
Nov 02 07:45:21.197367 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.197321    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-maas-api-service-ca" pod "1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" (UID: "1d755c19-a945-4f3b-97c5-f2b6bac1ee8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:21.210944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.210912    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-kube-api-access-jv95h" pod "1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" (UID: "1d755c19-a945-4f3b-97c5-f2b6bac1ee8c"). InnerVolumeSpecName "kube-api-access-jv95h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:21.227087 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.226685    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"
Nov 02 07:45:21.283957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.283607    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"
Nov 02 07:45:21.297342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.297313    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/234371f3-b3bf-4203-9dd3-d9dc57e666d0-kube-api-access-gs8bx\" (UniqueName: \"kubernetes.io/projected/234371f3-b3bf-4203-9dd3-d9dc57e666d0-kube-api-access-gs8bx\") pod \"234371f3-b3bf-4203-9dd3-d9dc57e666d0\" (UID: \"234371f3-b3bf-4203-9dd3-d9dc57e666d0\") "
Nov 02 07:45:21.297662 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.297389    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/234371f3-b3bf-4203-9dd3-d9dc57e666d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234371f3-b3bf-4203-9dd3-d9dc57e666d0-maas-api-service-ca\") pod \"234371f3-b3bf-4203-9dd3-d9dc57e666d0\" (UID: \"234371f3-b3bf-4203-9dd3-d9dc57e666d0\") "
Nov 02 07:45:21.297662 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.297533    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc34509-c942-498b-9324-ae77e0b22c03-kube-api-access-glm9g\" (UniqueName: \"kubernetes.io/projected/fdc34509-c942-498b-9324-ae77e0b22c03-kube-api-access-glm9g\") pod \"fdc34509-c942-498b-9324-ae77e0b22c03\" (UID: \"fdc34509-c942-498b-9324-ae77e0b22c03\") "
Nov 02 07:45:21.297825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.297672    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc34509-c942-498b-9324-ae77e0b22c03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc34509-c942-498b-9324-ae77e0b22c03-maas-api-service-ca\") pod \"fdc34509-c942-498b-9324-ae77e0b22c03\" (UID: \"fdc34509-c942-498b-9324-ae77e0b22c03\") "
Nov 02 07:45:21.298624 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.298373    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:21.298624 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.298400    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv95h\" (UniqueName: \"kubernetes.io/projected/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c-kube-api-access-jv95h\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:21.298795 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.298717    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc34509-c942-498b-9324-ae77e0b22c03-maas-api-service-ca" pod "fdc34509-c942-498b-9324-ae77e0b22c03" (UID: "fdc34509-c942-498b-9324-ae77e0b22c03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:21.299169 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.299141    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234371f3-b3bf-4203-9dd3-d9dc57e666d0-maas-api-service-ca" pod "234371f3-b3bf-4203-9dd3-d9dc57e666d0" (UID: "234371f3-b3bf-4203-9dd3-d9dc57e666d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:21.302664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.302603    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234371f3-b3bf-4203-9dd3-d9dc57e666d0-kube-api-access-gs8bx" pod "234371f3-b3bf-4203-9dd3-d9dc57e666d0" (UID: "234371f3-b3bf-4203-9dd3-d9dc57e666d0"). InnerVolumeSpecName "kube-api-access-gs8bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:21.303487 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.303462    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc34509-c942-498b-9324-ae77e0b22c03-kube-api-access-glm9g" pod "fdc34509-c942-498b-9324-ae77e0b22c03" (UID: "fdc34509-c942-498b-9324-ae77e0b22c03"). InnerVolumeSpecName "kube-api-access-glm9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:21.372245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.372214    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"
Nov 02 07:45:21.399792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.399761    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf4a75ea-2924-43ca-af92-1cf9cd51def8-kube-api-access-psxdj\" (UniqueName: \"kubernetes.io/projected/bf4a75ea-2924-43ca-af92-1cf9cd51def8-kube-api-access-psxdj\") pod \"bf4a75ea-2924-43ca-af92-1cf9cd51def8\" (UID: \"bf4a75ea-2924-43ca-af92-1cf9cd51def8\") "
Nov 02 07:45:21.399978 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.399812    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1e9db58-0abc-488f-8002-49ef49d20cac-kube-api-access-rmxws\" (UniqueName: \"kubernetes.io/projected/b1e9db58-0abc-488f-8002-49ef49d20cac-kube-api-access-rmxws\") pod \"b1e9db58-0abc-488f-8002-49ef49d20cac\" (UID: \"b1e9db58-0abc-488f-8002-49ef49d20cac\") "
Nov 02 07:45:21.399978 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.399929    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1e9db58-0abc-488f-8002-49ef49d20cac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e9db58-0abc-488f-8002-49ef49d20cac-maas-api-service-ca\") pod \"b1e9db58-0abc-488f-8002-49ef49d20cac\" (UID: \"b1e9db58-0abc-488f-8002-49ef49d20cac\") "
Nov 02 07:45:21.400178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.400138    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf4a75ea-2924-43ca-af92-1cf9cd51def8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4a75ea-2924-43ca-af92-1cf9cd51def8-maas-api-service-ca\") pod \"bf4a75ea-2924-43ca-af92-1cf9cd51def8\" (UID: \"bf4a75ea-2924-43ca-af92-1cf9cd51def8\") "
Nov 02 07:45:21.400752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.400489    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e9db58-0abc-488f-8002-49ef49d20cac-maas-api-service-ca" pod "b1e9db58-0abc-488f-8002-49ef49d20cac" (UID: "b1e9db58-0abc-488f-8002-49ef49d20cac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:21.400752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.400622    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4a75ea-2924-43ca-af92-1cf9cd51def8-maas-api-service-ca" pod "bf4a75ea-2924-43ca-af92-1cf9cd51def8" (UID: "bf4a75ea-2924-43ca-af92-1cf9cd51def8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:21.401416 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.401394    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4a75ea-2924-43ca-af92-1cf9cd51def8-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:21.401502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.401420    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs8bx\" (UniqueName: \"kubernetes.io/projected/234371f3-b3bf-4203-9dd3-d9dc57e666d0-kube-api-access-gs8bx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:21.401502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.401433    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234371f3-b3bf-4203-9dd3-d9dc57e666d0-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:21.401502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.401446    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e9db58-0abc-488f-8002-49ef49d20cac-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:21.401502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.401459    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glm9g\" (UniqueName: \"kubernetes.io/projected/fdc34509-c942-498b-9324-ae77e0b22c03-kube-api-access-glm9g\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:21.401502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.401473    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc34509-c942-498b-9324-ae77e0b22c03-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:21.403262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.403233    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e9db58-0abc-488f-8002-49ef49d20cac-kube-api-access-rmxws" pod "b1e9db58-0abc-488f-8002-49ef49d20cac" (UID: "b1e9db58-0abc-488f-8002-49ef49d20cac"). InnerVolumeSpecName "kube-api-access-rmxws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:21.404597 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.404551    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4a75ea-2924-43ca-af92-1cf9cd51def8-kube-api-access-psxdj" pod "bf4a75ea-2924-43ca-af92-1cf9cd51def8" (UID: "bf4a75ea-2924-43ca-af92-1cf9cd51def8"). InnerVolumeSpecName "kube-api-access-psxdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:21.474753 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.474721    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5325aca2-4bb7-4b29-9c83-fe4ce9bcd410" path="/var/lib/kubelet/pods/5325aca2-4bb7-4b29-9c83-fe4ce9bcd410/volumes"
Nov 02 07:45:21.475481 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.475460    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="626b6a54-25a8-4149-86f8-b2495d96b931" path="/var/lib/kubelet/pods/626b6a54-25a8-4149-86f8-b2495d96b931/volumes"
Nov 02 07:45:21.476053 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.476036    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82934468-ad39-4323-91c5-37163776131f" path="/var/lib/kubelet/pods/82934468-ad39-4323-91c5-37163776131f/volumes"
Nov 02 07:45:21.477189 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.477161    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7" path="/var/lib/kubelet/pods/b35036d5-6707-415a-ae82-ccd088c58cc7/volumes"
Nov 02 07:45:21.502871 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.502809    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psxdj\" (UniqueName: \"kubernetes.io/projected/bf4a75ea-2924-43ca-af92-1cf9cd51def8-kube-api-access-psxdj\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:21.502871 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.502841    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmxws\" (UniqueName: \"kubernetes.io/projected/b1e9db58-0abc-488f-8002-49ef49d20cac-kube-api-access-rmxws\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:21.746958 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.746925    4898 generic.go:356] "Generic (PLEG): container finished" podID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerID="bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7" exitCode=0
Nov 02 07:45:21.747375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.746995    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" event={"ID":"38fbf728-4e49-4574-9bc8-1d53e3356957","Type":"ContainerDied","Data":"bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7"}
Nov 02 07:45:21.747500 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.747480    4898 scope.go:122] "RemoveContainer" containerID="bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7"
Nov 02 07:45:21.748673 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.748650    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"
Nov 02 07:45:21.748673 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.748663    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf" event={"ID":"234371f3-b3bf-4203-9dd3-d9dc57e666d0","Type":"ContainerDied","Data":"d33f30c17a2751080e6b8fd55457ff15efba03007a1ad6bd2576b2494402ec0f"}
Nov 02 07:45:21.748834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.748721    4898 scope.go:122] "RemoveContainer" containerID="dfbfa61d7ccadff0c276adaa45fe2b53f60c56f0877bccd90b2a299391279d5b"
Nov 02 07:45:21.749225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.749189    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:21.749324 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.749286    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:21.749379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.749362    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:21.750312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.750291    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"
Nov 02 07:45:21.750412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.750320    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7" event={"ID":"fdc34509-c942-498b-9324-ae77e0b22c03","Type":"ContainerDied","Data":"97b64f48417006afbba3893c41f665b9781f59e501ad1efb2cfed39c040f33f0"}
Nov 02 07:45:21.752285 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.752261    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"
Nov 02 07:45:21.752406 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.752333    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t" event={"ID":"b1e9db58-0abc-488f-8002-49ef49d20cac","Type":"ContainerDied","Data":"72609ff7ac31f0acd1c0365b730437b08ffe3c11bd487d1be79d1b3fce8ce035"}
Nov 02 07:45:21.755518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.755493    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824" event={"ID":"1d755c19-a945-4f3b-97c5-f2b6bac1ee8c","Type":"ContainerDied","Data":"51ce3e39b42f90f688df15a29265c1ce00b1b45884aa4c57c837576f12a7f508"}
Nov 02 07:45:21.755618 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.755581    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"
Nov 02 07:45:21.758194 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.758180    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"
Nov 02 07:45:21.758403 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.758201    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9" event={"ID":"bf4a75ea-2924-43ca-af92-1cf9cd51def8","Type":"ContainerDied","Data":"c6c853e018c3b118cdd25939a2f32167ae62458ab3e3a4e527aeaaedc5833555"}
Nov 02 07:45:21.767471 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.767447    4898 scope.go:122] "RemoveContainer" containerID="3b683debc3a60089ed96d0d4132335243702b02d19be2c2078a41067ad905e22"
Nov 02 07:45:21.789291 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.789266    4898 scope.go:122] "RemoveContainer" containerID="4c4814f546bac716513665cd76f07889552dc52ab61557e20c9a143ef397df52"
Nov 02 07:45:21.806495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.806468    4898 scope.go:122] "RemoveContainer" containerID="4eb754dc3a8263f5ba8041b5b601ec9b2e169017e04d3a369ea08be47ad524c9"
Nov 02 07:45:21.827981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.827951    4898 scope.go:122] "RemoveContainer" containerID="4a214232c9d6067e13d9c2f9bd3d4999c456922e4e617bde56da438f80c3f8a5"
Nov 02 07:45:21.839663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.839638    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"]
Nov 02 07:45:21.846962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.846932    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893425-47824"]
Nov 02 07:45:21.881565 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.881532    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"]
Nov 02 07:45:21.889172 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.889117    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-77dc65fb-29893425-r2d4t"]
Nov 02 07:45:21.921496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.920995    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"]
Nov 02 07:45:21.925634 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.925609    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-6cc3f7-29893425-8sdg9"]
Nov 02 07:45:21.962836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.962759    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"]
Nov 02 07:45:21.970605 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.970047    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893425-57pjf"]
Nov 02 07:45:21.999971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:21.999834    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"]
Nov 02 07:45:22.005491 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:22.005451    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-523de0-29893425-nzpb7"]
Nov 02 07:45:22.795678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:22.795632    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" event={"ID":"38fbf728-4e49-4574-9bc8-1d53e3356957","Type":"ContainerStarted","Data":"43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f"}
Nov 02 07:45:22.977912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:22.968289    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:45:22.977912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:22.971988    4898 patch_prober.go:29] interesting pod/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 07:45:22.977912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:22.972041    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 07:45:23.040449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.040413    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"]
Nov 02 07:45:23.045389 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.045363    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c93d984-v7qvk"]
Nov 02 07:45:23.482913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.482730    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b6cf31-1501-4b1e-b5d0-bc4e391c8189" path="/var/lib/kubelet/pods/02b6cf31-1501-4b1e-b5d0-bc4e391c8189/volumes"
Nov 02 07:45:23.487743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.487458    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" path="/var/lib/kubelet/pods/1d755c19-a945-4f3b-97c5-f2b6bac1ee8c/volumes"
Nov 02 07:45:23.488044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.488027    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" path="/var/lib/kubelet/pods/234371f3-b3bf-4203-9dd3-d9dc57e666d0/volumes"
Nov 02 07:45:23.488591 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.488571    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac" path="/var/lib/kubelet/pods/b1e9db58-0abc-488f-8002-49ef49d20cac/volumes"
Nov 02 07:45:23.489676 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.489654    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" path="/var/lib/kubelet/pods/bf4a75ea-2924-43ca-af92-1cf9cd51def8/volumes"
Nov 02 07:45:23.490139 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.490115    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc34509-c942-498b-9324-ae77e0b22c03" path="/var/lib/kubelet/pods/fdc34509-c942-498b-9324-ae77e0b22c03/volumes"
Nov 02 07:45:23.744231 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.744128    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"]
Nov 02 07:45:23.745014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.744932    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fdc34509-c942-498b-9324-ae77e0b22c03" containerName="cleanup"
Nov 02 07:45:23.745014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.744958    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc34509-c942-498b-9324-ae77e0b22c03" containerName="cleanup"
Nov 02 07:45:23.745014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.744974    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" containerName="cleanup"
Nov 02 07:45:23.745014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.744982    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" containerName="cleanup"
Nov 02 07:45:23.745014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.744994    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7" containerName="cleanup"
Nov 02 07:45:23.745014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745002    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7" containerName="cleanup"
Nov 02 07:45:23.745014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745014    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fdc34509-c942-498b-9324-ae77e0b22c03" containerName="cleanup"
Nov 02 07:45:23.745014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745021    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc34509-c942-498b-9324-ae77e0b22c03" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745041    4898 cpu_manager.go:414] "containerMap: removing container" podUID="626b6a54-25a8-4149-86f8-b2495d96b931" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745048    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="626b6a54-25a8-4149-86f8-b2495d96b931" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745066    4898 cpu_manager.go:414] "containerMap: removing container" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745073    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745088    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745096    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745110    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745116    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745126    4898 cpu_manager.go:414] "containerMap: removing container" podUID="82934468-ad39-4323-91c5-37163776131f" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745132    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82934468-ad39-4323-91c5-37163776131f" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745141    4898 cpu_manager.go:414] "containerMap: removing container" podUID="626b6a54-25a8-4149-86f8-b2495d96b931" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745167    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="626b6a54-25a8-4149-86f8-b2495d96b931" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745182    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745189    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745207    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745213    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745228    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745236    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745248    4898 cpu_manager.go:414] "containerMap: removing container" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745255    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745281    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745289    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745299    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745306    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745319    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745326    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745337    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745344    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745409    4898 cpu_manager.go:414] "containerMap: removing container" podUID="82934468-ad39-4323-91c5-37163776131f" containerName="cleanup"
Nov 02 07:45:23.745516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745417    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82934468-ad39-4323-91c5-37163776131f" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745598    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="82934468-ad39-4323-91c5-37163776131f" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745610    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="82934468-ad39-4323-91c5-37163776131f" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745619    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745629    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745638    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="626b6a54-25a8-4149-86f8-b2495d96b931" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745681    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc34509-c942-498b-9324-ae77e0b22c03" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745691    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745700    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745709    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc34509-c942-498b-9324-ae77e0b22c03" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745720    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="626b6a54-25a8-4149-86f8-b2495d96b931" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745729    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745739    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745747    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="82934468-ad39-4323-91c5-37163776131f" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745758    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745767    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745776    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745784    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745799    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d755c19-a945-4f3b-97c5-f2b6bac1ee8c" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745809    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" containerName="cleanup"
Nov 02 07:45:23.746875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.745818    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4a75ea-2924-43ca-af92-1cf9cd51def8" containerName="cleanup"
Nov 02 07:45:23.751582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.751561    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.809544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.809498    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-54fd3c-openshift-default-dockercfg-25fhh\""
Nov 02 07:45:23.822090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.822061    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"]
Nov 02 07:45:23.875157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.875112    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-data\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.875435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.875269    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b71447-e15a-45e2-9230-4ec214621a65-istiod-ca-cert\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.875435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.875344    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-credential-socket\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.875435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.875395    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b71447-e15a-45e2-9230-4ec214621a65-istio-podinfo\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.875435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.875430    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-envoy\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.875709 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.875496    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-istio-token\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.875709 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.875539    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2jst\" (UniqueName: \"kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-kube-api-access-c2jst\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.875709 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.875607    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-socket\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.875709 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.875654    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-certs\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.964042 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.964004    4898 patch_prober.go:29] interesting pod/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 07:45:23.964399 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.964362    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 07:45:23.984915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.983010    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-data\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.984915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.983136    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b71447-e15a-45e2-9230-4ec214621a65-istiod-ca-cert\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.984915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.983198    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-credential-socket\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.984915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.983245    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b71447-e15a-45e2-9230-4ec214621a65-istio-podinfo\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.984915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.983282    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-envoy\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.984915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.983344    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-istio-token\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.984915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.983387    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2jst\" (UniqueName: \"kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-kube-api-access-c2jst\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.984915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.983466    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-socket\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.984915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.983516    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-certs\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.984915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.984089    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-certs\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:23.994969 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:23.987043    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-data\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:24.017912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.016028    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b71447-e15a-45e2-9230-4ec214621a65-istiod-ca-cert\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:24.017912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.016324    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-credential-socket\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:24.017912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.016864    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-socket\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:24.058914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.047748    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b71447-e15a-45e2-9230-4ec214621a65-istio-podinfo\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:24.058914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.048415    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-istio-token\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:24.058914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.055031    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-envoy\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:24.059851 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.059821    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2jst\" (UniqueName: \"kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-kube-api-access-c2jst\") pod \"e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") " pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:24.130916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.128293    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.411015    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"]
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412080    4898 cpu_manager.go:414] "containerMap: removing container" podUID="82934468-ad39-4323-91c5-37163776131f" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412105    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82934468-ad39-4323-91c5-37163776131f" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412125    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412133    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412225    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fdc34509-c942-498b-9324-ae77e0b22c03" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412233    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc34509-c942-498b-9324-ae77e0b22c03" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412251    4898 cpu_manager.go:414] "containerMap: removing container" podUID="626b6a54-25a8-4149-86f8-b2495d96b931" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412257    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="626b6a54-25a8-4149-86f8-b2495d96b931" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412273    4898 cpu_manager.go:414] "containerMap: removing container" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412280    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="234371f3-b3bf-4203-9dd3-d9dc57e666d0" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412438    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e9db58-0abc-488f-8002-49ef49d20cac" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412453    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc34509-c942-498b-9324-ae77e0b22c03" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412463    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="626b6a54-25a8-4149-86f8-b2495d96b931" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.412475    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35036d5-6707-415a-ae82-ccd088c58cc7" containerName="cleanup"
Nov 02 07:45:24.420911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.418209    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.499928 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.459552    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b2cc1f-openshift-default-dockercfg-jx7j5\""
Nov 02 07:45:24.563275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.561221    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-token\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.563275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.561277    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-envoy\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.563275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.561313    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-certs\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.563275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.561397    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-data\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.563275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.561426    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-podinfo\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.563275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.561742    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istiod-ca-cert\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.563275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.561789    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-socket\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.563275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.561873    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-credential-socket\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.563275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.561998    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l88hk\" (UniqueName: \"kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-kube-api-access-l88hk\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.635388 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.635347    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"]
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.670360    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-credential-socket\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.670441    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l88hk\" (UniqueName: \"kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-kube-api-access-l88hk\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.670497    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-token\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.670520    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-envoy\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.670549    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-certs\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.670599    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-data\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.670628    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-podinfo\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.670810    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istiod-ca-cert\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.670847    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-socket\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.671365    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-socket\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.671619    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-certs\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.671846    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-data\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.674594    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istiod-ca-cert\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.701496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.675331    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-credential-socket\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.704743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.703974    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l88hk\" (UniqueName: \"kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-kube-api-access-l88hk\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.704743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.704665    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-token\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.711765 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.709973    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-podinfo\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.711765 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.710156    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-envoy\") pod \"e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") " pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.949464 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.943006    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"]
Nov 02 07:45:24.954417 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.953640    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:24.968902 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.968848    4898 patch_prober.go:29] interesting pod/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 07:45:24.971959 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.971922    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 07:45:24.986134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.986088    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:24.986515 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.986451    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:24.986720 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:24.986694    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:25.209963 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:25.205765    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:45:25.421601 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:45:25.421573    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b859db2_8e95_4eab_8ebc_fb0e3d26b532.slice/crio-5dac18b4036cc5f8fb3321851e7932f4cb597d756d419de3aef19301c3104b8b WatchSource:0}: Error finding container 5dac18b4036cc5f8fb3321851e7932f4cb597d756d419de3aef19301c3104b8b: Status 404 returned error can't find the container with id 5dac18b4036cc5f8fb3321851e7932f4cb597d756d419de3aef19301c3104b8b
Nov 02 07:45:25.424543 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:25.424507    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:25.424714 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:25.424697    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:25.424811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:25.424798    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:25.425561 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:25.425539    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"]
Nov 02 07:45:25.921820 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:25.921778    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" event={"ID":"a5b71447-e15a-45e2-9230-4ec214621a65","Type":"ContainerStarted","Data":"3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97"}
Nov 02 07:45:25.922034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:25.921840    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" event={"ID":"a5b71447-e15a-45e2-9230-4ec214621a65","Type":"ContainerStarted","Data":"148400a9931dc916efae343ff956448886557542b719b5ddc83ea78bb24de100"}
Nov 02 07:45:25.924789 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:25.924758    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" event={"ID":"2b859db2-8e95-4eab-8ebc-fb0e3d26b532","Type":"ContainerStarted","Data":"e5c1491d64ec98a8aa1b9e455614dd884ec260cf89591f035dac2943bc10ff98"}
Nov 02 07:45:25.924917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:25.924813    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" event={"ID":"2b859db2-8e95-4eab-8ebc-fb0e3d26b532","Type":"ContainerStarted","Data":"5dac18b4036cc5f8fb3321851e7932f4cb597d756d419de3aef19301c3104b8b"}
Nov 02 07:45:25.954431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:25.954403    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:25.968915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:25.967870    4898 patch_prober.go:29] interesting pod/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 07:45:25.968915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:25.967948    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 07:45:26.087530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:26.087479    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" podStartSLOduration=3.087463868 podStartE2EDuration="3.087463868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:26.000846828 +0000 UTC m=+1276.691173792" watchObservedRunningTime="2026-11-02 07:45:26.087463868 +0000 UTC m=+1276.777790832"
Nov 02 07:45:26.090864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:26.090820    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" podStartSLOduration=2.09080674 podStartE2EDuration="2.09080674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:26.086524557 +0000 UTC m=+1276.776851544" watchObservedRunningTime="2026-11-02 07:45:26.09080674 +0000 UTC m=+1276.781133757"
Nov 02 07:45:26.130149 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:26.130120    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:26.136093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:26.136070    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:27.021913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:27.019111    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:27.021913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:27.019442    4898 patch_prober.go:29] interesting pod/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 07:45:27.021913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:27.019485    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 07:45:27.023495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:27.023437    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:45:27.080262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:27.080227    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:27.607567 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:27.607527    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 07:45:27.969004 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:27.968936    4898 patch_prober.go:29] interesting pod/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 02 07:45:27.970453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:27.970411    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 02 07:45:28.037915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:28.035962    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:28.037915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:28.037059    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:45:28.667397 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:28.667367    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"
Nov 02 07:45:28.966482 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:28.966409    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:45:28.967692 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:28.967667    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:45:29.195914 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:29.195122    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 07:45:29.201935 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:29.201848    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 07:45:30.644995 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:30.644953    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"]
Nov 02 07:45:30.722914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:30.707142    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"
Nov 02 07:45:30.761350 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:30.743212    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"]
Nov 02 07:45:30.761350 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:30.751145    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-54fd3c-dockercfg-zq9jz\""
Nov 02 07:45:30.761350 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:30.751334    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-54fd3c\""
Nov 02 07:45:30.801956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:30.800039    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898cbf78-6478-41a2-937d-30f8e5da22f2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct\" (UID: \"898cbf78-6478-41a2-937d-30f8e5da22f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"
Nov 02 07:45:30.801956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:30.800497    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nzxh\" (UniqueName: \"kubernetes.io/projected/898cbf78-6478-41a2-937d-30f8e5da22f2-kube-api-access-2nzxh\") pod \"payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct\" (UID: \"898cbf78-6478-41a2-937d-30f8e5da22f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"
Nov 02 07:45:30.832339 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:30.832307    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 07:45:30.902590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:30.902552    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nzxh\" (UniqueName: \"kubernetes.io/projected/898cbf78-6478-41a2-937d-30f8e5da22f2-kube-api-access-2nzxh\") pod \"payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct\" (UID: \"898cbf78-6478-41a2-937d-30f8e5da22f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"
Nov 02 07:45:30.902795 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:30.902660    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898cbf78-6478-41a2-937d-30f8e5da22f2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct\" (UID: \"898cbf78-6478-41a2-937d-30f8e5da22f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"
Nov 02 07:45:30.903593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:30.903562    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898cbf78-6478-41a2-937d-30f8e5da22f2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct\" (UID: \"898cbf78-6478-41a2-937d-30f8e5da22f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"
Nov 02 07:45:30.926951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:30.926877    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nzxh\" (UniqueName: \"kubernetes.io/projected/898cbf78-6478-41a2-937d-30f8e5da22f2-kube-api-access-2nzxh\") pod \"payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct\" (UID: \"898cbf78-6478-41a2-937d-30f8e5da22f2\") " pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"
Nov 02 07:45:31.058661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:31.058632    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"
Nov 02 07:45:31.366946 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:45:31.366855    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898cbf78_6478_41a2_937d_30f8e5da22f2.slice/crio-771304b88c89b34894f5f255557ed466f8f78e0534a331e78acc305f07b24153 WatchSource:0}: Error finding container 771304b88c89b34894f5f255557ed466f8f78e0534a331e78acc305f07b24153: Status 404 returned error can't find the container with id 771304b88c89b34894f5f255557ed466f8f78e0534a331e78acc305f07b24153
Nov 02 07:45:31.369219 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:31.369152    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"]
Nov 02 07:45:32.035533 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.035497    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"]
Nov 02 07:45:32.047443 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.047413    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c78ed9d-4wxwl"]
Nov 02 07:45:32.075173 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.075138    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct" event={"ID":"898cbf78-6478-41a2-937d-30f8e5da22f2","Type":"ContainerStarted","Data":"f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a"}
Nov 02 07:45:32.075311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.075203    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct" event={"ID":"898cbf78-6478-41a2-937d-30f8e5da22f2","Type":"ContainerStarted","Data":"771304b88c89b34894f5f255557ed466f8f78e0534a331e78acc305f07b24153"}
Nov 02 07:45:32.075311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.075232    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"
Nov 02 07:45:32.116276 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.116228    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct" podStartSLOduration=2.1162161 podStartE2EDuration="2.1162161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:32.115445926 +0000 UTC m=+1282.805772889" watchObservedRunningTime="2026-11-02 07:45:32.1162161 +0000 UTC m=+1282.806543101"
Nov 02 07:45:32.565910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.490342    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"]
Nov 02 07:45:32.664053 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.661701    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"]
Nov 02 07:45:32.664053 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.661854    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"
Nov 02 07:45:32.686456 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.686412    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b2cc1f-dockercfg-pq7nk\""
Nov 02 07:45:32.756315 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.756264    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff0ab99-f59f-4b01-a2f5-704b760da63f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp\" (UID: \"6ff0ab99-f59f-4b01-a2f5-704b760da63f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"
Nov 02 07:45:32.756620 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.756424    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtptg\" (UniqueName: \"kubernetes.io/projected/6ff0ab99-f59f-4b01-a2f5-704b760da63f-kube-api-access-dtptg\") pod \"payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp\" (UID: \"6ff0ab99-f59f-4b01-a2f5-704b760da63f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"
Nov 02 07:45:32.757506 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.757480    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"]
Nov 02 07:45:32.763168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.763144    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.768630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.768606    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b2cc1f\""
Nov 02 07:45:32.792679 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.792645    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"]
Nov 02 07:45:32.793292 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.793266    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-dc62c2-openshift-default-dockercfg-9r5jz\""
Nov 02 07:45:32.865733 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.865645    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-envoy\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.865733 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.865715    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-socket\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.866141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.865756    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-token\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.866525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.866498    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff0ab99-f59f-4b01-a2f5-704b760da63f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp\" (UID: \"6ff0ab99-f59f-4b01-a2f5-704b760da63f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"
Nov 02 07:45:32.866597 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.866535    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfdsl\" (UniqueName: \"kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-kube-api-access-zfdsl\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.866597 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.866558    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-credential-socket\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.866704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.866603    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-data\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.866704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.866643    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-certs\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.866704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.866674    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtptg\" (UniqueName: \"kubernetes.io/projected/6ff0ab99-f59f-4b01-a2f5-704b760da63f-kube-api-access-dtptg\") pod \"payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp\" (UID: \"6ff0ab99-f59f-4b01-a2f5-704b760da63f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"
Nov 02 07:45:32.866704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.866691    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81861f8c-9af4-4530-b75a-6f8180c8f003-istiod-ca-cert\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.866952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.866751    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-podinfo\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.867628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.867597    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff0ab99-f59f-4b01-a2f5-704b760da63f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp\" (UID: \"6ff0ab99-f59f-4b01-a2f5-704b760da63f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"
Nov 02 07:45:32.914593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.914555    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtptg\" (UniqueName: \"kubernetes.io/projected/6ff0ab99-f59f-4b01-a2f5-704b760da63f-kube-api-access-dtptg\") pod \"payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp\" (UID: \"6ff0ab99-f59f-4b01-a2f5-704b760da63f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"
Nov 02 07:45:32.967928 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.967877    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdsl\" (UniqueName: \"kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-kube-api-access-zfdsl\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.968104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.967938    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-credential-socket\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.968104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.967998    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-data\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.968104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.968067    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-certs\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.968294 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.968124    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81861f8c-9af4-4530-b75a-6f8180c8f003-istiod-ca-cert\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.968294 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.968254    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-podinfo\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.968446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.968412    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-credential-socket\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.968516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.968460    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-envoy\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.968516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.968467    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-data\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.968630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.968516    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-socket\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.968630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.968550    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-certs\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.968630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.968576    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-token\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.968836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.968770    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-socket\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.969046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.969005    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81861f8c-9af4-4530-b75a-6f8180c8f003-istiod-ca-cert\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.970781 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.970753    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-podinfo\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.971045 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.971006    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-envoy\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.993202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.993175    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdsl\" (UniqueName: \"kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-kube-api-access-zfdsl\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:32.993373 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:32.993354    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-token\") pod \"e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") " pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:33.026670 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:33.026645    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"
Nov 02 07:45:33.085474 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:33.085428    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:33.433537 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:45:33.433509    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff0ab99_f59f_4b01_a2f5_704b760da63f.slice/crio-0cbd407febbe23ad6c54456e1de3f385a004e060c5d7c424108c3ced5c0d59d6 WatchSource:0}: Error finding container 0cbd407febbe23ad6c54456e1de3f385a004e060c5d7c424108c3ced5c0d59d6: Status 404 returned error can't find the container with id 0cbd407febbe23ad6c54456e1de3f385a004e060c5d7c424108c3ced5c0d59d6
Nov 02 07:45:33.436239 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:33.436213    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"]
Nov 02 07:45:33.469346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:33.469320    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ec3fd5-230c-4ce6-b093-814e6ef74954" path="/var/lib/kubelet/pods/70ec3fd5-230c-4ce6-b093-814e6ef74954/volumes"
Nov 02 07:45:33.534424 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:45:33.534400    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81861f8c_9af4_4530_b75a_6f8180c8f003.slice/crio-759715032d8706266b5c45c2b721dbc2f89fe8aa683397c5dcb727383e37b0a1 WatchSource:0}: Error finding container 759715032d8706266b5c45c2b721dbc2f89fe8aa683397c5dcb727383e37b0a1: Status 404 returned error can't find the container with id 759715032d8706266b5c45c2b721dbc2f89fe8aa683397c5dcb727383e37b0a1
Nov 02 07:45:33.537548 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:33.537513    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:33.537646 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:33.537584    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:33.537646 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:33.537613    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:45:33.538195 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:33.538175    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"]
Nov 02 07:45:34.031492 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:34.031458    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"]
Nov 02 07:45:34.036181 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:34.036144    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1022cc-7xr76"]
Nov 02 07:45:34.122312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:34.122272    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp" event={"ID":"6ff0ab99-f59f-4b01-a2f5-704b760da63f","Type":"ContainerStarted","Data":"e90f018d2bc6d9f7f5998f4697478a15027a68f90a682985f080d7e724279346"}
Nov 02 07:45:34.122690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:34.122337    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp" event={"ID":"6ff0ab99-f59f-4b01-a2f5-704b760da63f","Type":"ContainerStarted","Data":"0cbd407febbe23ad6c54456e1de3f385a004e060c5d7c424108c3ced5c0d59d6"}
Nov 02 07:45:34.122690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:34.122620    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"
Nov 02 07:45:34.124123 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:34.124099    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" event={"ID":"81861f8c-9af4-4530-b75a-6f8180c8f003","Type":"ContainerStarted","Data":"b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a"}
Nov 02 07:45:34.124299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:34.124145    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" event={"ID":"81861f8c-9af4-4530-b75a-6f8180c8f003","Type":"ContainerStarted","Data":"759715032d8706266b5c45c2b721dbc2f89fe8aa683397c5dcb727383e37b0a1"}
Nov 02 07:45:34.179255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:34.179190    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp" podStartSLOduration=2.179171975 podStartE2EDuration="2.179171975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:34.178401252 +0000 UTC m=+1284.868728258" watchObservedRunningTime="2026-11-02 07:45:34.179171975 +0000 UTC m=+1284.869498942"
Nov 02 07:45:34.234268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:34.234195    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" podStartSLOduration=2.234177084 podStartE2EDuration="2.234177084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:45:34.230049109 +0000 UTC m=+1284.920376090" watchObservedRunningTime="2026-11-02 07:45:34.234177084 +0000 UTC m=+1284.924504047"
Nov 02 07:45:35.097227 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:35.097188    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:35.114946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:35.113337    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:35.181911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:35.180961    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:35.236265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:35.235906    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:45:35.467827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:35.467788    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90794722-dc5a-4d3c-9756-5584bb04d93c" path="/var/lib/kubelet/pods/90794722-dc5a-4d3c-9756-5584bb04d93c/volumes"
Nov 02 07:45:38.039782 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:38.039748    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"]
Nov 02 07:45:38.050245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:38.050213    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e40a3a13-nxb7b"]
Nov 02 07:45:39.468161 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:39.468109    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96fc6a9-7bc9-4a19-be83-b8fa78a650d0" path="/var/lib/kubelet/pods/b96fc6a9-7bc9-4a19-be83-b8fa78a650d0/volumes"
Nov 02 07:45:40.923865 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:40.923829    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 07:45:42.531756 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:42.531718    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 07:45:43.115640 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:43.115611    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"
Nov 02 07:45:44.657824 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:44.657793    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"]
Nov 02 07:45:44.658290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:44.658024    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" containerName="istio-proxy" containerID="cri-o://344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca" gracePeriod=30
Nov 02 07:45:44.996994 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:44.994606    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:45:45.184848 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:45.184820    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"
Nov 02 07:45:47.034336 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:47.034294    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"]
Nov 02 07:45:47.046204 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:47.046165    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d79ea55-x7b92"]
Nov 02 07:45:47.467253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:47.467215    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb779ab5-b3d7-4bfa-a842-92b1ce6be750" path="/var/lib/kubelet/pods/bb779ab5-b3d7-4bfa-a842-92b1ce6be750/volumes"
Nov 02 07:45:48.038854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:48.036839    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"]
Nov 02 07:45:48.046845 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:48.046812    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f5e02d-pwkvj"]
Nov 02 07:45:49.468527 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:49.468491    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386bd924-0c5b-453a-aa47-8afd6bff971f" path="/var/lib/kubelet/pods/386bd924-0c5b-453a-aa47-8afd6bff971f/volumes"
Nov 02 07:45:49.967513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:49.967483    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:45:50.010203 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010169    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-credential-socket\") pod \"f0f85612-ef12-4dad-826f-3711b7f72703\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") "
Nov 02 07:45:50.010396 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010256    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-envoy\") pod \"f0f85612-ef12-4dad-826f-3711b7f72703\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") "
Nov 02 07:45:50.010396 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010295    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-istio-token\" (UniqueName: \"kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-istio-token\") pod \"f0f85612-ef12-4dad-826f-3711b7f72703\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") "
Nov 02 07:45:50.010396 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010349    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-data\") pod \"f0f85612-ef12-4dad-826f-3711b7f72703\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") "
Nov 02 07:45:50.010396 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010364    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-certs\") pod \"f0f85612-ef12-4dad-826f-3711b7f72703\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") "
Nov 02 07:45:50.010628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010439    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-socket\") pod \"f0f85612-ef12-4dad-826f-3711b7f72703\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") "
Nov 02 07:45:50.010628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010467    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0f85612-ef12-4dad-826f-3711b7f72703-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f85612-ef12-4dad-826f-3711b7f72703-istio-podinfo\") pod \"f0f85612-ef12-4dad-826f-3711b7f72703\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") "
Nov 02 07:45:50.010628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010503    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-kube-api-access-6lx9g\" (UniqueName: \"kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-kube-api-access-6lx9g\") pod \"f0f85612-ef12-4dad-826f-3711b7f72703\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") "
Nov 02 07:45:50.010628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010462    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-credential-socket" pod "f0f85612-ef12-4dad-826f-3711b7f72703" (UID: "f0f85612-ef12-4dad-826f-3711b7f72703"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:45:50.010836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010640    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-data" pod "f0f85612-ef12-4dad-826f-3711b7f72703" (UID: "f0f85612-ef12-4dad-826f-3711b7f72703"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:45:50.010836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010658    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f85612-ef12-4dad-826f-3711b7f72703-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f85612-ef12-4dad-826f-3711b7f72703-istiod-ca-cert\") pod \"f0f85612-ef12-4dad-826f-3711b7f72703\" (UID: \"f0f85612-ef12-4dad-826f-3711b7f72703\") "
Nov 02 07:45:50.010836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010728    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-socket" pod "f0f85612-ef12-4dad-826f-3711b7f72703" (UID: "f0f85612-ef12-4dad-826f-3711b7f72703"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:45:50.010836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010748    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-certs" pod "f0f85612-ef12-4dad-826f-3711b7f72703" (UID: "f0f85612-ef12-4dad-826f-3711b7f72703"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:45:50.011146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.010976    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f85612-ef12-4dad-826f-3711b7f72703-istiod-ca-cert" pod "f0f85612-ef12-4dad-826f-3711b7f72703" (UID: "f0f85612-ef12-4dad-826f-3711b7f72703"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:45:50.011571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.011545    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:50.011681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.011577    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:50.011681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.011592    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:50.011681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.011605    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f85612-ef12-4dad-826f-3711b7f72703-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:50.011681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.011620    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:50.012664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.012636    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-istio-token" pod "f0f85612-ef12-4dad-826f-3711b7f72703" (UID: "f0f85612-ef12-4dad-826f-3711b7f72703"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:50.012771 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.012744    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-envoy" pod "f0f85612-ef12-4dad-826f-3711b7f72703" (UID: "f0f85612-ef12-4dad-826f-3711b7f72703"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:45:50.012825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.012805    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0f85612-ef12-4dad-826f-3711b7f72703-istio-podinfo" pod "f0f85612-ef12-4dad-826f-3711b7f72703" (UID: "f0f85612-ef12-4dad-826f-3711b7f72703"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:45:50.012864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.012822    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-kube-api-access-6lx9g" pod "f0f85612-ef12-4dad-826f-3711b7f72703" (UID: "f0f85612-ef12-4dad-826f-3711b7f72703"). InnerVolumeSpecName "kube-api-access-6lx9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:45:50.112978 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.112879    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f85612-ef12-4dad-826f-3711b7f72703-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:50.112978 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.112930    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:50.112978 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.112944    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f85612-ef12-4dad-826f-3711b7f72703-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:50.112978 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.112953    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lx9g\" (UniqueName: \"kubernetes.io/projected/f0f85612-ef12-4dad-826f-3711b7f72703-kube-api-access-6lx9g\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:45:50.357941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.349108    4898 generic.go:356] "Generic (PLEG): container finished" podID="f0f85612-ef12-4dad-826f-3711b7f72703" containerID="344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca" exitCode=0
Nov 02 07:45:50.357941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.349544    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" event={"ID":"f0f85612-ef12-4dad-826f-3711b7f72703","Type":"ContainerDied","Data":"344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca"}
Nov 02 07:45:50.357941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.349615    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk" event={"ID":"f0f85612-ef12-4dad-826f-3711b7f72703","Type":"ContainerDied","Data":"4634ee7585b3fdad536679629a52e8dce1d03e1a07592e77926032a9abfafa59"}
Nov 02 07:45:50.357941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.349652    4898 scope.go:122] "RemoveContainer" containerID="344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca"
Nov 02 07:45:50.357941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.349830    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"
Nov 02 07:45:50.421869 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.421848    4898 scope.go:122] "RemoveContainer" containerID="229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5"
Nov 02 07:45:50.432825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.432800    4898 scope.go:122] "RemoveContainer" containerID="344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca"
Nov 02 07:45:50.433149 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:50.433122    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca\": container with ID starting with 344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca not found: ID does not exist" containerID="344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca"
Nov 02 07:45:50.433245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.433162    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca"} err="failed to get container status \"344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca\": rpc error: code = NotFound desc = could not find container \"344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca\": container with ID starting with 344b5905a52b4015dfd38968e0cef0f7a348269ce30a4ddd55d5dd181973dcca not found: ID does not exist"
Nov 02 07:45:50.433245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.433186    4898 scope.go:122] "RemoveContainer" containerID="229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5"
Nov 02 07:45:50.433471 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:50.433453    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5\": container with ID starting with 229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5 not found: ID does not exist" containerID="229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5"
Nov 02 07:45:50.433514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.433478    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5"} err="failed to get container status \"229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5\": rpc error: code = NotFound desc = could not find container \"229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5\": container with ID starting with 229476af4eadb2688e762b86ef99b3610d2e877cbcad6e22087082ed85f556d5 not found: ID does not exist"
Nov 02 07:45:50.443949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.443811    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"]
Nov 02 07:45:50.451584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:50.451544    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-77dc65fb-openshift-default-b6b44599c-htvdk"]
Nov 02 07:45:50.940344 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:50.940305    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 07:45:51.466621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:51.466583    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" path="/var/lib/kubelet/pods/f0f85612-ef12-4dad-826f-3711b7f72703/volumes"
Nov 02 07:45:54.034314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:54.034280    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"]
Nov 02 07:45:54.046452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:54.046416    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfb47050-d8b6l"]
Nov 02 07:45:55.465951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:45:55.465913    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e909df00-b3ca-42cd-ac3a-1db5a9872fdd" path="/var/lib/kubelet/pods/e909df00-b3ca-42cd-ac3a-1db5a9872fdd/volumes"
Nov 02 07:45:57.575727 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:45:57.575694    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 07:46:00.960795 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:46:00.960751    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a5ca3d_fa04_4acd_b5b9_f02bfd52dbbf.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 07:46:06.034624 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:06.034585    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"]
Nov 02 07:46:06.046062 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:06.046027    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"]
Nov 02 07:46:06.050289 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:06.050271    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4c5c758-xk7hc"]
Nov 02 07:46:06.054024 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:06.054000    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f856dfed-q9vk7"]
Nov 02 07:46:06.528728 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:06.528691    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"]
Nov 02 07:46:06.529001 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:06.528972    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn" podUID="262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1" containerName="payload-processing" containerID="cri-o://1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a" gracePeriod=30
Nov 02 07:46:06.852813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:06.852787    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"
Nov 02 07:46:06.899938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:06.899881    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-plugins-config-volume\") pod \"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1\" (UID: \"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1\") "
Nov 02 07:46:06.900136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:06.900056    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-kube-api-access-s2xsn\" (UniqueName: \"kubernetes.io/projected/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-kube-api-access-s2xsn\") pod \"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1\" (UID: \"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1\") "
Nov 02 07:46:06.911932 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:06.911476    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-plugins-config-volume" pod "262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1" (UID: "262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:06.924212 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:06.924126    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-kube-api-access-s2xsn" pod "262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1" (UID: "262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1"). InnerVolumeSpecName "kube-api-access-s2xsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:07.018933 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.012995    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:07.018933 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.013033    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2xsn\" (UniqueName: \"kubernetes.io/projected/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1-kube-api-access-s2xsn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:07.481058 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.480999    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c3d3a8-727e-403e-ac99-3841f6b43ad0" path="/var/lib/kubelet/pods/54c3d3a8-727e-403e-ac99-3841f6b43ad0/volumes"
Nov 02 07:46:07.482518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.482490    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2df4ed-fb73-4014-995e-eddd01ae1c18" path="/var/lib/kubelet/pods/ec2df4ed-fb73-4014-995e-eddd01ae1c18/volumes"
Nov 02 07:46:07.520367 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.520330    4898 generic.go:356] "Generic (PLEG): container finished" podID="262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1" containerID="1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a" exitCode=0
Nov 02 07:46:07.520546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.520393    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"
Nov 02 07:46:07.520546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.520413    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn" event={"ID":"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1","Type":"ContainerDied","Data":"1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a"}
Nov 02 07:46:07.520546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.520473    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn" event={"ID":"262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1","Type":"ContainerDied","Data":"d15084725ced2dbc208aeb8259e2c84e6ded6c28ea86ccc905d2578ee042140c"}
Nov 02 07:46:07.520546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.520503    4898 scope.go:122] "RemoveContainer" containerID="1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a"
Nov 02 07:46:07.534084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.533929    4898 scope.go:122] "RemoveContainer" containerID="1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a"
Nov 02 07:46:07.534241 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:46:07.534223    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a\": container with ID starting with 1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a not found: ID does not exist" containerID="1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a"
Nov 02 07:46:07.534288 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.534250    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a"} err="failed to get container status \"1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a\": rpc error: code = NotFound desc = could not find container \"1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a\": container with ID starting with 1047220750fd3142bdfa79750c9986ddeb3777baa63e23409041cc00e2f0fa7a not found: ID does not exist"
Nov 02 07:46:07.576290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.576257    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"]
Nov 02 07:46:07.588328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:07.588300    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e07c6-85cffcc7fd-gh2rn"]
Nov 02 07:46:09.468458 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:09.468424    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1" path="/var/lib/kubelet/pods/262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1/volumes"
Nov 02 07:46:10.222470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:10.222434    4898 scope.go:122] "RemoveContainer" containerID="c171a739726425d209be672699f0a94c4488a8df35bfec18b8b3806f5b49aa29"
Nov 02 07:46:10.238634 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:10.238613    4898 scope.go:122] "RemoveContainer" containerID="bb46e37522ab362d12554c7543ec5b12f00cfd24c075e48a0410420543e64f24"
Nov 02 07:46:10.252380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:10.252353    4898 scope.go:122] "RemoveContainer" containerID="5ae622b894ded9b6055fb218c77d21a863e35825ead4b1bab08dc99ccac773c4"
Nov 02 07:46:10.264299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:10.264275    4898 scope.go:122] "RemoveContainer" containerID="07e0edf66187f3b2d3402488e0454f60c392fc969f4fcee2ef8ba2cd09e4bf65"
Nov 02 07:46:10.276425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:10.276402    4898 scope.go:122] "RemoveContainer" containerID="625a055bce0408c76cde64bbd5bc2eb310edbfbe6b75b975668a94b167ce62e7"
Nov 02 07:46:10.294572 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:10.294546    4898 scope.go:122] "RemoveContainer" containerID="03f61866c92e9a4520ce82ad5d7348bea7f7dad00b120c20508e54e0f278bf47"
Nov 02 07:46:10.306961 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:10.306878    4898 scope.go:122] "RemoveContainer" containerID="26d20e5cda9383ac33df5f57328ee0e20af61db0b7805833776f16aca92aa3d8"
Nov 02 07:46:10.316452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:10.316431    4898 scope.go:122] "RemoveContainer" containerID="1392aa99e4c2ad6a9bf94280928f619760dab6716e3dffd0ef674bf3c65511cd"
Nov 02 07:46:10.329608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:10.329531    4898 scope.go:122] "RemoveContainer" containerID="b3e7cd04f47d478cd6a4e4c335025e8e50e7c8603d0bebf14bf27860e7fa22d0"
Nov 02 07:46:10.477037 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:10.468169    4898 scope.go:122] "RemoveContainer" containerID="7a75bb67920e0be74b802d7c0b97efb8e7123bff9b42d96e41329970d8dfd674"
Nov 02 07:46:10.561173 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:10.557191    4898 scope.go:122] "RemoveContainer" containerID="9fe9f6fff63144d2d309a40717bf918f93ce2abb44c40d7255268a7870b370de"
Nov 02 07:46:14.990419 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:46:14.990377    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:46:28.063082 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:28.063027    4898 patch_prober.go:29] interesting pod/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:46:28.063690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:28.063118    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:46:29.078089 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:29.078041    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:46:29.078848 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:29.078112    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:46:29.602655 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:29.602621    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"]
Nov 02 07:46:34.603578 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:34.603533    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp" podUID="6ff0ab99-f59f-4b01-a2f5-704b760da63f" containerName="payload-processing" containerID="cri-o://e90f018d2bc6d9f7f5998f4697478a15027a68f90a682985f080d7e724279346" gracePeriod=25
Nov 02 07:46:35.160642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:35.160604    4898 generic.go:356] "Generic (PLEG): container finished" podID="6ff0ab99-f59f-4b01-a2f5-704b760da63f" containerID="e90f018d2bc6d9f7f5998f4697478a15027a68f90a682985f080d7e724279346" exitCode=0
Nov 02 07:46:35.160898 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:35.160866    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp" event={"ID":"6ff0ab99-f59f-4b01-a2f5-704b760da63f","Type":"ContainerDied","Data":"e90f018d2bc6d9f7f5998f4697478a15027a68f90a682985f080d7e724279346"}
Nov 02 07:46:35.240103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:35.240077    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"
Nov 02 07:46:35.318460 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:35.318411    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff0ab99-f59f-4b01-a2f5-704b760da63f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff0ab99-f59f-4b01-a2f5-704b760da63f-plugins-config-volume\") pod \"6ff0ab99-f59f-4b01-a2f5-704b760da63f\" (UID: \"6ff0ab99-f59f-4b01-a2f5-704b760da63f\") "
Nov 02 07:46:35.318624 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:35.318524    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff0ab99-f59f-4b01-a2f5-704b760da63f-kube-api-access-dtptg\" (UniqueName: \"kubernetes.io/projected/6ff0ab99-f59f-4b01-a2f5-704b760da63f-kube-api-access-dtptg\") pod \"6ff0ab99-f59f-4b01-a2f5-704b760da63f\" (UID: \"6ff0ab99-f59f-4b01-a2f5-704b760da63f\") "
Nov 02 07:46:35.318849 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:35.318823    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff0ab99-f59f-4b01-a2f5-704b760da63f-plugins-config-volume" pod "6ff0ab99-f59f-4b01-a2f5-704b760da63f" (UID: "6ff0ab99-f59f-4b01-a2f5-704b760da63f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:35.319206 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:35.319182    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff0ab99-f59f-4b01-a2f5-704b760da63f-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:35.320779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:35.320756    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff0ab99-f59f-4b01-a2f5-704b760da63f-kube-api-access-dtptg" pod "6ff0ab99-f59f-4b01-a2f5-704b760da63f" (UID: "6ff0ab99-f59f-4b01-a2f5-704b760da63f"). InnerVolumeSpecName "kube-api-access-dtptg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:35.420616 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:35.420582    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtptg\" (UniqueName: \"kubernetes.io/projected/6ff0ab99-f59f-4b01-a2f5-704b760da63f-kube-api-access-dtptg\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:36.173470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:36.173419    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp" event={"ID":"6ff0ab99-f59f-4b01-a2f5-704b760da63f","Type":"ContainerDied","Data":"0cbd407febbe23ad6c54456e1de3f385a004e060c5d7c424108c3ced5c0d59d6"}
Nov 02 07:46:36.173470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:36.173463    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"
Nov 02 07:46:36.459237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:36.173509    4898 scope.go:122] "RemoveContainer" containerID="e90f018d2bc6d9f7f5998f4697478a15027a68f90a682985f080d7e724279346"
Nov 02 07:46:36.459237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:36.181838    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp" podUID="6ff0ab99-f59f-4b01-a2f5-704b760da63f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.160:50052\" within 1s: context deadline exceeded"
Nov 02 07:46:36.459237 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:46:36.181909    4898 logging.go:55] [core] [Channel #1454 SubChannel #1455]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.160:50052", ServerName: "10.132.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.160:50052: operation was canceled"
Nov 02 07:46:36.459237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:36.219991    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:46:36.459237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:36.220044    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:46:36.459237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:36.230941    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"]
Nov 02 07:46:36.459237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:36.235693    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2cc1f-d4b4d7cdf-55lrp"]
Nov 02 07:46:37.184739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.184706    4898 generic.go:356] "Generic (PLEG): container finished" podID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerID="b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a" exitCode=0
Nov 02 07:46:37.185247 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.184787    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" event={"ID":"81861f8c-9af4-4530-b75a-6f8180c8f003","Type":"ContainerDied","Data":"b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a"}
Nov 02 07:46:37.185397 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.185379    4898 scope.go:122] "RemoveContainer" containerID="b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a"
Nov 02 07:46:37.187379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.187344    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:46:37.187500 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.187437    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:46:37.187500 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.187483    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:46:37.187795 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.187776    4898 generic.go:356] "Generic (PLEG): container finished" podID="a5b71447-e15a-45e2-9230-4ec214621a65" containerID="3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97" exitCode=0
Nov 02 07:46:37.187872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.187853    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" event={"ID":"a5b71447-e15a-45e2-9230-4ec214621a65","Type":"ContainerDied","Data":"3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97"}
Nov 02 07:46:37.188292 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.188264    4898 scope.go:122] "RemoveContainer" containerID="3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97"
Nov 02 07:46:37.189883 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.189850    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:46:37.189999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.189988    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:46:37.190065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.190021    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:46:37.191160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.191137    4898 generic.go:356] "Generic (PLEG): container finished" podID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerID="e5c1491d64ec98a8aa1b9e455614dd884ec260cf89591f035dac2943bc10ff98" exitCode=0
Nov 02 07:46:37.191299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.191283    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" event={"ID":"2b859db2-8e95-4eab-8ebc-fb0e3d26b532","Type":"ContainerDied","Data":"e5c1491d64ec98a8aa1b9e455614dd884ec260cf89591f035dac2943bc10ff98"}
Nov 02 07:46:37.191672 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.191655    4898 scope.go:122] "RemoveContainer" containerID="e5c1491d64ec98a8aa1b9e455614dd884ec260cf89591f035dac2943bc10ff98"
Nov 02 07:46:37.193551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.193523    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:46:37.193615 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.193599    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:46:37.193660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.193645    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:46:37.482264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:37.482186    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff0ab99-f59f-4b01-a2f5-704b760da63f" path="/var/lib/kubelet/pods/6ff0ab99-f59f-4b01-a2f5-704b760da63f/volumes"
Nov 02 07:46:38.281861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:38.280484    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" event={"ID":"81861f8c-9af4-4530-b75a-6f8180c8f003","Type":"ContainerStarted","Data":"bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba"}
Nov 02 07:46:38.286299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:38.286265    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" event={"ID":"a5b71447-e15a-45e2-9230-4ec214621a65","Type":"ContainerStarted","Data":"4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47"}
Nov 02 07:46:38.311125 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:38.311090    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" event={"ID":"2b859db2-8e95-4eab-8ebc-fb0e3d26b532","Type":"ContainerStarted","Data":"1003b2e05258003e59509a784df5ba85f90ebd1abde144d7c0a88db0ae171e2e"}
Nov 02 07:46:39.016939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.005653    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:46:39.029934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.027080    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 07:46:39.029934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.027142    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 07:46:39.091380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.091350    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:46:39.105912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.105642    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 07:46:39.105912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.105706    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 07:46:39.140913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.133291    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:46:39.140913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.133438    4898 patch_prober.go:29] interesting pod/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 07:46:39.140913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.133477    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 07:46:39.557077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.557033    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:46:39.676010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.675963    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:46:39.954105 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.954075    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 07:46:39.954344 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:39.954322    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 07:46:40.090626 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:40.090587    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 07:46:40.090923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:40.090874    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 07:46:40.130378 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:40.130344    4898 patch_prober.go:29] interesting pod/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 07:46:40.130650 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:40.130619    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 07:46:40.984912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:40.973342    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 07:46:40.984912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:40.973421    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 07:46:41.092914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:41.092835    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 07:46:41.093102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:41.092936    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 07:46:41.140923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:41.140326    4898 patch_prober.go:29] interesting pod/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 07:46:41.140923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:41.140408    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 07:46:41.979934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:41.955267    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 07:46:41.979934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:41.955335    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 07:46:42.020962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:42.020921    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:46:42.121098 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:42.119216    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 07:46:42.121098 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:42.119378    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 07:46:42.133292 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:42.133192    4898 patch_prober.go:29] interesting pod/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 07:46:42.133292 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:42.133245    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 07:46:43.022924 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:43.015001    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 07:46:43.030036 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:43.029996    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 07:46:43.060242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:43.060207    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:46:43.151729 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:43.151690    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 07:46:43.152051 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:43.151985    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 07:46:43.178921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:43.155916    4898 patch_prober.go:29] interesting pod/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 07:46:43.178921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:43.155958    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 07:46:43.969915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:43.962874    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 07:46:43.969915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:43.962962    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 07:46:44.121913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:44.114836    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 07:46:44.121913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:44.114927    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 07:46:44.133425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:44.133391    4898 patch_prober.go:29] interesting pod/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 07:46:44.133665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:44.133637    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 07:46:44.960472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:44.960437    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:46:44.963346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:44.963319    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:46:45.007820 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:46:45.007775    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:46:45.139845 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:45.139812    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:46:45.142013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:45.141988    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:46:45.224406 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:45.224268    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:46:45.225202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:45.225176    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:46:45.227736 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:45.227716    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:46:46.913537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:46.913499    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"]
Nov 02 07:46:46.914261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:46.914227    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy" containerID="cri-o://43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f" gracePeriod=30
Nov 02 07:46:52.246849 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.246819    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:46:52.328393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.328358    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-certs\") pod \"38fbf728-4e49-4574-9bc8-1d53e3356957\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") "
Nov 02 07:46:52.328671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.328652    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-kube-api-access-fkhp6\" (UniqueName: \"kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-kube-api-access-fkhp6\") pod \"38fbf728-4e49-4574-9bc8-1d53e3356957\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") "
Nov 02 07:46:52.328776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.328647    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-certs" pod "38fbf728-4e49-4574-9bc8-1d53e3356957" (UID: "38fbf728-4e49-4574-9bc8-1d53e3356957"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:52.328776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.328713    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38fbf728-4e49-4574-9bc8-1d53e3356957-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbf728-4e49-4574-9bc8-1d53e3356957-istiod-ca-cert\") pod \"38fbf728-4e49-4574-9bc8-1d53e3356957\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") "
Nov 02 07:46:52.328884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.328834    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-credential-socket\") pod \"38fbf728-4e49-4574-9bc8-1d53e3356957\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") "
Nov 02 07:46:52.328970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.328928    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-data\") pod \"38fbf728-4e49-4574-9bc8-1d53e3356957\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") "
Nov 02 07:46:52.329107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.328989    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-token\" (UniqueName: \"kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-token\") pod \"38fbf728-4e49-4574-9bc8-1d53e3356957\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") "
Nov 02 07:46:52.329107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.329041    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-socket\") pod \"38fbf728-4e49-4574-9bc8-1d53e3356957\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") "
Nov 02 07:46:52.329210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.329129    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-envoy\") pod \"38fbf728-4e49-4574-9bc8-1d53e3356957\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") "
Nov 02 07:46:52.329263 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.329240    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-podinfo\") pod \"38fbf728-4e49-4574-9bc8-1d53e3356957\" (UID: \"38fbf728-4e49-4574-9bc8-1d53e3356957\") "
Nov 02 07:46:52.329337 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.329307    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fbf728-4e49-4574-9bc8-1d53e3356957-istiod-ca-cert" pod "38fbf728-4e49-4574-9bc8-1d53e3356957" (UID: "38fbf728-4e49-4574-9bc8-1d53e3356957"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:52.329578 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.329553    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-socket" pod "38fbf728-4e49-4574-9bc8-1d53e3356957" (UID: "38fbf728-4e49-4574-9bc8-1d53e3356957"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:52.329785 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.329765    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-data" pod "38fbf728-4e49-4574-9bc8-1d53e3356957" (UID: "38fbf728-4e49-4574-9bc8-1d53e3356957"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:52.330037 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.329990    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-credential-socket" pod "38fbf728-4e49-4574-9bc8-1d53e3356957" (UID: "38fbf728-4e49-4574-9bc8-1d53e3356957"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:52.331069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.331047    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:52.331217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.331202    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:52.331320 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.331309    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbf728-4e49-4574-9bc8-1d53e3356957-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:52.331411 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.331399    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:52.331492 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.331481    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:52.333600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.333575    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-envoy" pod "38fbf728-4e49-4574-9bc8-1d53e3356957" (UID: "38fbf728-4e49-4574-9bc8-1d53e3356957"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:46:52.333686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.333642    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-token" pod "38fbf728-4e49-4574-9bc8-1d53e3356957" (UID: "38fbf728-4e49-4574-9bc8-1d53e3356957"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:52.338341 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.338316    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-podinfo" pod "38fbf728-4e49-4574-9bc8-1d53e3356957" (UID: "38fbf728-4e49-4574-9bc8-1d53e3356957"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:46:52.338439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.338395    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-kube-api-access-fkhp6" pod "38fbf728-4e49-4574-9bc8-1d53e3356957" (UID: "38fbf728-4e49-4574-9bc8-1d53e3356957"). InnerVolumeSpecName "kube-api-access-fkhp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:52.433900 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.433868    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:52.434047 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.433910    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkhp6\" (UniqueName: \"kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-kube-api-access-fkhp6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:52.434047 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.433924    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:52.434047 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:52.433937    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbf728-4e49-4574-9bc8-1d53e3356957-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:53.043069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.043033    4898 generic.go:356] "Generic (PLEG): container finished" podID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerID="43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f" exitCode=0
Nov 02 07:46:53.043267 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.043085    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"
Nov 02 07:46:53.043267 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.043116    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" event={"ID":"38fbf728-4e49-4574-9bc8-1d53e3356957","Type":"ContainerDied","Data":"43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f"}
Nov 02 07:46:53.043267 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.043179    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv" event={"ID":"38fbf728-4e49-4574-9bc8-1d53e3356957","Type":"ContainerDied","Data":"cd5e9e07b779f9a69cdb8dd504d04c256eef5d9e6c68b8a1d674ec2e99288c80"}
Nov 02 07:46:53.043267 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.043214    4898 scope.go:122] "RemoveContainer" containerID="43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f"
Nov 02 07:46:53.063141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.058180    4898 scope.go:122] "RemoveContainer" containerID="bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7"
Nov 02 07:46:53.077067 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.077041    4898 scope.go:122] "RemoveContainer" containerID="43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f"
Nov 02 07:46:53.077426 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:46:53.077398    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f\": container with ID starting with 43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f not found: ID does not exist" containerID="43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f"
Nov 02 07:46:53.077525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.077434    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f"} err="failed to get container status \"43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f\": rpc error: code = NotFound desc = could not find container \"43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f\": container with ID starting with 43d24d32fc373156fc303e5f790317f356cb847c029df5ed7b53cf235231203f not found: ID does not exist"
Nov 02 07:46:53.077525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.077453    4898 scope.go:122] "RemoveContainer" containerID="bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7"
Nov 02 07:46:53.077781 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:46:53.077761    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7\": container with ID starting with bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7 not found: ID does not exist" containerID="bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7"
Nov 02 07:46:53.077854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.077786    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7"} err="failed to get container status \"bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7\": rpc error: code = NotFound desc = could not find container \"bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7\": container with ID starting with bd5835fc0777ec94dcd9cc0dee6ef62279a76c56cfba642a783d33827a746db7 not found: ID does not exist"
Nov 02 07:46:53.111745 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.111708    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"]
Nov 02 07:46:53.124215 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.124172    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5a032f-openshift-default-5bb6d5df59-5fpmv"]
Nov 02 07:46:53.468666 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.468631    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" path="/var/lib/kubelet/pods/38fbf728-4e49-4574-9bc8-1d53e3356957/volumes"
Nov 02 07:46:53.558812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:53.558779    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"]
Nov 02 07:46:58.559301 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:58.559235    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct" podUID="898cbf78-6478-41a2-937d-30f8e5da22f2" containerName="payload-processing" containerID="cri-o://f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a" gracePeriod=25
Nov 02 07:46:58.914820 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:58.914793    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"
Nov 02 07:46:58.989637 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:58.989601    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/898cbf78-6478-41a2-937d-30f8e5da22f2-kube-api-access-2nzxh\" (UniqueName: \"kubernetes.io/projected/898cbf78-6478-41a2-937d-30f8e5da22f2-kube-api-access-2nzxh\") pod \"898cbf78-6478-41a2-937d-30f8e5da22f2\" (UID: \"898cbf78-6478-41a2-937d-30f8e5da22f2\") "
Nov 02 07:46:58.990006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:58.989980    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/898cbf78-6478-41a2-937d-30f8e5da22f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898cbf78-6478-41a2-937d-30f8e5da22f2-plugins-config-volume\") pod \"898cbf78-6478-41a2-937d-30f8e5da22f2\" (UID: \"898cbf78-6478-41a2-937d-30f8e5da22f2\") "
Nov 02 07:46:58.990628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:58.990503    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898cbf78-6478-41a2-937d-30f8e5da22f2-plugins-config-volume" pod "898cbf78-6478-41a2-937d-30f8e5da22f2" (UID: "898cbf78-6478-41a2-937d-30f8e5da22f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:46:58.991417 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:58.991390    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898cbf78-6478-41a2-937d-30f8e5da22f2-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:58.992414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:58.992391    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898cbf78-6478-41a2-937d-30f8e5da22f2-kube-api-access-2nzxh" pod "898cbf78-6478-41a2-937d-30f8e5da22f2" (UID: "898cbf78-6478-41a2-937d-30f8e5da22f2"). InnerVolumeSpecName "kube-api-access-2nzxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:46:59.093226 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:59.093192    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nzxh\" (UniqueName: \"kubernetes.io/projected/898cbf78-6478-41a2-937d-30f8e5da22f2-kube-api-access-2nzxh\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:46:59.456184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:59.456140    4898 generic.go:356] "Generic (PLEG): container finished" podID="898cbf78-6478-41a2-937d-30f8e5da22f2" containerID="f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a" exitCode=0
Nov 02 07:46:59.456399 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:59.456193    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"
Nov 02 07:46:59.456399 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:59.456218    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct" event={"ID":"898cbf78-6478-41a2-937d-30f8e5da22f2","Type":"ContainerDied","Data":"f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a"}
Nov 02 07:46:59.456399 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:59.456350    4898 scope.go:122] "RemoveContainer" containerID="f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a"
Nov 02 07:46:59.469420 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:59.469388    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct" event={"ID":"898cbf78-6478-41a2-937d-30f8e5da22f2","Type":"ContainerDied","Data":"771304b88c89b34894f5f255557ed466f8f78e0534a331e78acc305f07b24153"}
Nov 02 07:46:59.472199 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:59.472172    4898 scope.go:122] "RemoveContainer" containerID="f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a"
Nov 02 07:46:59.472514 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:46:59.472490    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a\": container with ID starting with f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a not found: ID does not exist" containerID="f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a"
Nov 02 07:46:59.472596 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:59.472526    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a"} err="failed to get container status \"f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a\": rpc error: code = NotFound desc = could not find container \"f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a\": container with ID starting with f90cc1b9bf7e7080f7765ad770d7f4e465547bfcc47876d4dcadc3f60372e85a not found: ID does not exist"
Nov 02 07:46:59.518225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:59.518189    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"]
Nov 02 07:46:59.529994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:46:59.529968    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-54fd3c-7644ff887-l97ct"]
Nov 02 07:47:00.030397 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:00.030361    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"]
Nov 02 07:47:00.042540 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:00.042509    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddba9d62-g6shf"]
Nov 02 07:47:01.466347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:01.466310    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="898cbf78-6478-41a2-937d-30f8e5da22f2" path="/var/lib/kubelet/pods/898cbf78-6478-41a2-937d-30f8e5da22f2/volumes"
Nov 02 07:47:01.466832 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:01.466816    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e91ed93d-6a2c-40e2-9bc8-b46de087e9e4" path="/var/lib/kubelet/pods/e91ed93d-6a2c-40e2-9bc8-b46de087e9e4/volumes"
Nov 02 07:47:10.773507 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:10.773473    4898 scope.go:122] "RemoveContainer" containerID="ca5f19351a7358568c596e020db193907fc3d7b67b4b2b818d393fb90bdc821e"
Nov 02 07:47:10.786308 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:10.786279    4898 scope.go:122] "RemoveContainer" containerID="1a4cc0214e91b58a4bcee979323abc4be4916bcff9793d5741e3c2dc46c970a7"
Nov 02 07:47:10.796471 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:10.796444    4898 scope.go:122] "RemoveContainer" containerID="3a79323281ba38d773dd6490149f56bfb334e94c4276ae1190030aa0bea894c1"
Nov 02 07:47:10.807936 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:10.807873    4898 scope.go:122] "RemoveContainer" containerID="c3ac452f7d0f0e7e667395b2ae13a6acecbb0e75d2a6d11e41ec40364f46a4de"
Nov 02 07:47:10.817533 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:10.817410    4898 scope.go:122] "RemoveContainer" containerID="14d3f8d805631699924c37220646604ba12535b01542fa9bd479e15b4abb8646"
Nov 02 07:47:10.829517 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:10.829488    4898 scope.go:122] "RemoveContainer" containerID="0996037d27715c806d8a6550a6b6385bdd12e78d1bd85f05338001c781460d56"
Nov 02 07:47:10.838649 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:10.838632    4898 scope.go:122] "RemoveContainer" containerID="c85f71663c920f103db8da6931395e779d1ee16442aa56d8af872def72fa0ec2"
Nov 02 07:47:10.847540 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:10.847519    4898 scope.go:122] "RemoveContainer" containerID="c13a9909cacfb600e554f12145d240345b0a91434c440ffad8e17bcc44e96460"
Nov 02 07:47:10.856874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:10.856848    4898 scope.go:122] "RemoveContainer" containerID="4d03392d7d624db0cf8fe9698e0ba20f898aa8e161ebff4d315d6503f0988bbb"
Nov 02 07:47:10.866323 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:10.866300    4898 scope.go:122] "RemoveContainer" containerID="baccb7d67f44ae37f0dea7d29fae07769b7442ddca0468127d7a725fe5c50894"
Nov 02 07:47:15.052127 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:47:15.052082    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:47:15.616612 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:15.616578    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"]
Nov 02 07:47:15.616832 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:15.616802    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl" podUID="07f3f205-27d6-4401-be74-e655ca60c83a" containerName="payload-processing" containerID="cri-o://51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f" gracePeriod=30
Nov 02 07:47:15.641955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:15.641459    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"]
Nov 02 07:47:15.641955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:15.641735    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784" podUID="2be61eaf-5961-4f71-a125-037d47ac2c54" containerName="payload-pre-processing" containerID="cri-o://37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d" gracePeriod=30
Nov 02 07:47:16.267315 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.267286    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"
Nov 02 07:47:16.271827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.271804    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"
Nov 02 07:47:16.383993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.383961    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f3f205-27d6-4401-be74-e655ca60c83a-kube-api-access-xdllx\" (UniqueName: \"kubernetes.io/projected/07f3f205-27d6-4401-be74-e655ca60c83a-kube-api-access-xdllx\") pod \"07f3f205-27d6-4401-be74-e655ca60c83a\" (UID: \"07f3f205-27d6-4401-be74-e655ca60c83a\") "
Nov 02 07:47:16.384195 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.384005    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f3f205-27d6-4401-be74-e655ca60c83a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f3f205-27d6-4401-be74-e655ca60c83a-plugins-config-volume\") pod \"07f3f205-27d6-4401-be74-e655ca60c83a\" (UID: \"07f3f205-27d6-4401-be74-e655ca60c83a\") "
Nov 02 07:47:16.384195 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.384055    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be61eaf-5961-4f71-a125-037d47ac2c54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be61eaf-5961-4f71-a125-037d47ac2c54-plugins-config-volume\") pod \"2be61eaf-5961-4f71-a125-037d47ac2c54\" (UID: \"2be61eaf-5961-4f71-a125-037d47ac2c54\") "
Nov 02 07:47:16.384346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.384325    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be61eaf-5961-4f71-a125-037d47ac2c54-kube-api-access-jhkb7\" (UniqueName: \"kubernetes.io/projected/2be61eaf-5961-4f71-a125-037d47ac2c54-kube-api-access-jhkb7\") pod \"2be61eaf-5961-4f71-a125-037d47ac2c54\" (UID: \"2be61eaf-5961-4f71-a125-037d47ac2c54\") "
Nov 02 07:47:16.384618 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.384590    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f3f205-27d6-4401-be74-e655ca60c83a-plugins-config-volume" pod "07f3f205-27d6-4401-be74-e655ca60c83a" (UID: "07f3f205-27d6-4401-be74-e655ca60c83a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:16.392476 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.384850    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be61eaf-5961-4f71-a125-037d47ac2c54-plugins-config-volume" pod "2be61eaf-5961-4f71-a125-037d47ac2c54" (UID: "2be61eaf-5961-4f71-a125-037d47ac2c54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:16.392673 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.387817    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be61eaf-5961-4f71-a125-037d47ac2c54-kube-api-access-jhkb7" pod "2be61eaf-5961-4f71-a125-037d47ac2c54" (UID: "2be61eaf-5961-4f71-a125-037d47ac2c54"). InnerVolumeSpecName "kube-api-access-jhkb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:16.392818 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.387831    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f3f205-27d6-4401-be74-e655ca60c83a-kube-api-access-xdllx" pod "07f3f205-27d6-4401-be74-e655ca60c83a" (UID: "07f3f205-27d6-4401-be74-e655ca60c83a"). InnerVolumeSpecName "kube-api-access-xdllx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:16.487241 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.487152    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhkb7\" (UniqueName: \"kubernetes.io/projected/2be61eaf-5961-4f71-a125-037d47ac2c54-kube-api-access-jhkb7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:16.487241 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.487185    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdllx\" (UniqueName: \"kubernetes.io/projected/07f3f205-27d6-4401-be74-e655ca60c83a-kube-api-access-xdllx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:16.487241 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.487197    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f3f205-27d6-4401-be74-e655ca60c83a-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:16.487241 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.487209    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be61eaf-5961-4f71-a125-037d47ac2c54-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:16.658968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.658930    4898 generic.go:356] "Generic (PLEG): container finished" podID="2be61eaf-5961-4f71-a125-037d47ac2c54" containerID="37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d" exitCode=0
Nov 02 07:47:16.659149 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.659017    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"
Nov 02 07:47:16.659149 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.659022    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784" event={"ID":"2be61eaf-5961-4f71-a125-037d47ac2c54","Type":"ContainerDied","Data":"37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d"}
Nov 02 07:47:16.659276 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.659165    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784" event={"ID":"2be61eaf-5961-4f71-a125-037d47ac2c54","Type":"ContainerDied","Data":"a5df95acdbca984efed7aed99194af32b17d635b1157ad3e90af96b43f83c2ba"}
Nov 02 07:47:16.659276 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.659231    4898 scope.go:122] "RemoveContainer" containerID="37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d"
Nov 02 07:47:16.664187 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.664147    4898 generic.go:356] "Generic (PLEG): container finished" podID="07f3f205-27d6-4401-be74-e655ca60c83a" containerID="51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f" exitCode=0
Nov 02 07:47:16.664372 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.664275    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"
Nov 02 07:47:16.664372 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.664311    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl" event={"ID":"07f3f205-27d6-4401-be74-e655ca60c83a","Type":"ContainerDied","Data":"51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f"}
Nov 02 07:47:16.664372 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.664357    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl" event={"ID":"07f3f205-27d6-4401-be74-e655ca60c83a","Type":"ContainerDied","Data":"0a1c03dba1a0df8adbdb4ba06ac2b3ebc05361c64579deb96b78dd02184b4a3b"}
Nov 02 07:47:16.679240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.679216    4898 scope.go:122] "RemoveContainer" containerID="37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d"
Nov 02 07:47:16.679935 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:47:16.679870    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d\": container with ID starting with 37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d not found: ID does not exist" containerID="37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d"
Nov 02 07:47:16.680047 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.679947    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d"} err="failed to get container status \"37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d\": rpc error: code = NotFound desc = could not find container \"37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d\": container with ID starting with 37cf0db4af722820b56c29295ec50ad030036685213d0947a262e5ed3a14bb3d not found: ID does not exist"
Nov 02 07:47:16.680047 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.679972    4898 scope.go:122] "RemoveContainer" containerID="51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f"
Nov 02 07:47:16.693392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.693369    4898 scope.go:122] "RemoveContainer" containerID="51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f"
Nov 02 07:47:16.693698 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:47:16.693668    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f\": container with ID starting with 51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f not found: ID does not exist" containerID="51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f"
Nov 02 07:47:16.693788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.693706    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f"} err="failed to get container status \"51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f\": rpc error: code = NotFound desc = could not find container \"51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f\": container with ID starting with 51cba540650f87284d0c021d2bf4042b74bdf141a1cae2a41194530ac59f954f not found: ID does not exist"
Nov 02 07:47:16.723731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.723702    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"]
Nov 02 07:47:16.729182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.729144    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-523de0-896768d7f-m8784"]
Nov 02 07:47:16.770185 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.770084    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"]
Nov 02 07:47:16.776945 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:16.776912    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-523de0-78f4cd5c96-w5dgl"]
Nov 02 07:47:17.466582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:17.466547    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f3f205-27d6-4401-be74-e655ca60c83a" path="/var/lib/kubelet/pods/07f3f205-27d6-4401-be74-e655ca60c83a/volumes"
Nov 02 07:47:17.467223 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:17.467201    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be61eaf-5961-4f71-a125-037d47ac2c54" path="/var/lib/kubelet/pods/2be61eaf-5961-4f71-a125-037d47ac2c54/volumes"
Nov 02 07:47:17.722763 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:17.722675    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"]
Nov 02 07:47:17.723325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:17.723289    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy" containerID="cri-o://4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47" gracePeriod=30
Nov 02 07:47:18.363947 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:18.363909    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"]
Nov 02 07:47:18.364556 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:18.364517    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz" podUID="b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" containerName="istio-proxy" containerID="cri-o://672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973" gracePeriod=30
Nov 02 07:47:23.156292 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.156262    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:47:23.191736 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.191649    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-credential-socket\") pod \"a5b71447-e15a-45e2-9230-4ec214621a65\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") "
Nov 02 07:47:23.191900 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.191765    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5b71447-e15a-45e2-9230-4ec214621a65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b71447-e15a-45e2-9230-4ec214621a65-istio-podinfo\") pod \"a5b71447-e15a-45e2-9230-4ec214621a65\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") "
Nov 02 07:47:23.191900 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.191879    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-data\") pod \"a5b71447-e15a-45e2-9230-4ec214621a65\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") "
Nov 02 07:47:23.192001 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.191933    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-certs\") pod \"a5b71447-e15a-45e2-9230-4ec214621a65\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") "
Nov 02 07:47:23.192056 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.191991    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-credential-socket" pod "a5b71447-e15a-45e2-9230-4ec214621a65" (UID: "a5b71447-e15a-45e2-9230-4ec214621a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.192113 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.192090    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-envoy\") pod \"a5b71447-e15a-45e2-9230-4ec214621a65\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") "
Nov 02 07:47:23.192175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.192165    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-socket\") pod \"a5b71447-e15a-45e2-9230-4ec214621a65\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") "
Nov 02 07:47:23.192229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.192191    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-data" pod "a5b71447-e15a-45e2-9230-4ec214621a65" (UID: "a5b71447-e15a-45e2-9230-4ec214621a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.192229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.192207    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b71447-e15a-45e2-9230-4ec214621a65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b71447-e15a-45e2-9230-4ec214621a65-istiod-ca-cert\") pod \"a5b71447-e15a-45e2-9230-4ec214621a65\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") "
Nov 02 07:47:23.192345 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.192219    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-certs" pod "a5b71447-e15a-45e2-9230-4ec214621a65" (UID: "a5b71447-e15a-45e2-9230-4ec214621a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.192345 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.192268    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-istio-token\" (UniqueName: \"kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-istio-token\") pod \"a5b71447-e15a-45e2-9230-4ec214621a65\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") "
Nov 02 07:47:23.192345 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.192304    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-kube-api-access-c2jst\" (UniqueName: \"kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-kube-api-access-c2jst\") pod \"a5b71447-e15a-45e2-9230-4ec214621a65\" (UID: \"a5b71447-e15a-45e2-9230-4ec214621a65\") "
Nov 02 07:47:23.192503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.192452    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-socket" pod "a5b71447-e15a-45e2-9230-4ec214621a65" (UID: "a5b71447-e15a-45e2-9230-4ec214621a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.192574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.192553    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b71447-e15a-45e2-9230-4ec214621a65-istiod-ca-cert" pod "a5b71447-e15a-45e2-9230-4ec214621a65" (UID: "a5b71447-e15a-45e2-9230-4ec214621a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:23.193184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.193157    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.193307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.193213    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.193307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.193233    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.193307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.193247    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.193307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.193262    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b71447-e15a-45e2-9230-4ec214621a65-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.194821 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.194782    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-envoy" pod "a5b71447-e15a-45e2-9230-4ec214621a65" (UID: "a5b71447-e15a-45e2-9230-4ec214621a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.195003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.194977    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-istio-token" pod "a5b71447-e15a-45e2-9230-4ec214621a65" (UID: "a5b71447-e15a-45e2-9230-4ec214621a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:23.195448 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.195421    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-kube-api-access-c2jst" pod "a5b71447-e15a-45e2-9230-4ec214621a65" (UID: "a5b71447-e15a-45e2-9230-4ec214621a65"). InnerVolumeSpecName "kube-api-access-c2jst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:23.195761 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.195744    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5b71447-e15a-45e2-9230-4ec214621a65-istio-podinfo" pod "a5b71447-e15a-45e2-9230-4ec214621a65" (UID: "a5b71447-e15a-45e2-9230-4ec214621a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:47:23.294975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.294939    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b71447-e15a-45e2-9230-4ec214621a65-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.294975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.294968    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.294975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.294984    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2jst\" (UniqueName: \"kubernetes.io/projected/a5b71447-e15a-45e2-9230-4ec214621a65-kube-api-access-c2jst\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.295232 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.294994    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b71447-e15a-45e2-9230-4ec214621a65-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.650228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.650203    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:47:23.701488 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.701450    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istiod-ca-cert\") pod \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") "
Nov 02 07:47:23.701704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.701508    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-envoy\") pod \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") "
Nov 02 07:47:23.701704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.701546    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-credential-socket\") pod \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") "
Nov 02 07:47:23.701704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.701577    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-data\") pod \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") "
Nov 02 07:47:23.701704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.701634    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-kube-api-access-zkddr\" (UniqueName: \"kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-kube-api-access-zkddr\") pod \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") "
Nov 02 07:47:23.701704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.701663    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-token\" (UniqueName: \"kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-token\") pod \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") "
Nov 02 07:47:23.702083 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.701788    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-socket\") pod \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") "
Nov 02 07:47:23.702083 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.702060    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-certs\") pod \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") "
Nov 02 07:47:23.702187 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.702093    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-podinfo\") pod \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\" (UID: \"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226\") "
Nov 02 07:47:23.702881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.702336    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-credential-socket" pod "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" (UID: "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.702881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.702537    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-socket" pod "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" (UID: "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.702881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.702550    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-data" pod "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" (UID: "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.702881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.702748    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-certs" pod "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" (UID: "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.702881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.702871    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.703225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.702914    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.703225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.702932    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.703225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.702945    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.703225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.703108    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istiod-ca-cert" pod "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" (UID: "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:23.707296 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.707269    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-token" pod "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" (UID: "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:23.709164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.709137    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-kube-api-access-zkddr" pod "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" (UID: "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226"). InnerVolumeSpecName "kube-api-access-zkddr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:23.709306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.709269    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-envoy" pod "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" (UID: "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:23.748578 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.746107    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-podinfo" pod "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" (UID: "b5f2b49f-9601-41b6-b9e2-5cbab0c8d226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:47:23.818236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.817953    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.818236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.817989    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.818236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.818002    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.818236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.818016    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkddr\" (UniqueName: \"kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-kube-api-access-zkddr\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.818236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.818030    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:23.825250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.825218    4898 generic.go:356] "Generic (PLEG): container finished" podID="b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" containerID="672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973" exitCode=0
Nov 02 07:47:23.825414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.825271    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"
Nov 02 07:47:23.825414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.825302    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz" event={"ID":"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226","Type":"ContainerDied","Data":"672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973"}
Nov 02 07:47:23.825414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.825362    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz" event={"ID":"b5f2b49f-9601-41b6-b9e2-5cbab0c8d226","Type":"ContainerDied","Data":"1dee96bebf5dd25d2ce2f13075d86b25e4162cf09f25a2ef13882d3d5649e497"}
Nov 02 07:47:23.825414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.825384    4898 scope.go:122] "RemoveContainer" containerID="672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973"
Nov 02 07:47:23.828457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.828419    4898 generic.go:356] "Generic (PLEG): container finished" podID="a5b71447-e15a-45e2-9230-4ec214621a65" containerID="4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47" exitCode=0
Nov 02 07:47:23.828581 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.828462    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" event={"ID":"a5b71447-e15a-45e2-9230-4ec214621a65","Type":"ContainerDied","Data":"4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47"}
Nov 02 07:47:23.828581 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.828470    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"
Nov 02 07:47:23.828581 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.828501    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd" event={"ID":"a5b71447-e15a-45e2-9230-4ec214621a65","Type":"ContainerDied","Data":"148400a9931dc916efae343ff956448886557542b719b5ddc83ea78bb24de100"}
Nov 02 07:47:23.838622 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.838605    4898 scope.go:122] "RemoveContainer" containerID="672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973"
Nov 02 07:47:23.838908 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:47:23.838877    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973\": container with ID starting with 672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973 not found: ID does not exist" containerID="672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973"
Nov 02 07:47:23.838988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.838921    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973"} err="failed to get container status \"672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973\": rpc error: code = NotFound desc = could not find container \"672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973\": container with ID starting with 672a259319d30fded128fd48365755b5f54df925df178b47f9a27c5f27270973 not found: ID does not exist"
Nov 02 07:47:23.838988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.838945    4898 scope.go:122] "RemoveContainer" containerID="4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47"
Nov 02 07:47:23.851556 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.851502    4898 scope.go:122] "RemoveContainer" containerID="3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97"
Nov 02 07:47:23.868059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.864965    4898 scope.go:122] "RemoveContainer" containerID="4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47"
Nov 02 07:47:23.868059 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:47:23.867046    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47\": container with ID starting with 4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47 not found: ID does not exist" containerID="4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47"
Nov 02 07:47:23.868059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.867091    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47"} err="failed to get container status \"4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47\": rpc error: code = NotFound desc = could not find container \"4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47\": container with ID starting with 4de7db4387b713d8904c074c6c69f67d8c17f80e447e1a781b5b31c68679fa47 not found: ID does not exist"
Nov 02 07:47:23.868059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.867119    4898 scope.go:122] "RemoveContainer" containerID="3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97"
Nov 02 07:47:23.875229 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:47:23.875194    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97\": container with ID starting with 3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97 not found: ID does not exist" containerID="3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97"
Nov 02 07:47:23.875349 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.875238    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97"} err="failed to get container status \"3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97\": rpc error: code = NotFound desc = could not find container \"3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97\": container with ID starting with 3da3e09ea7d95d9e9e329768883d4328a2d8522849dd0d0483b334ab7fd69c97 not found: ID does not exist"
Nov 02 07:47:23.889218 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.889185    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"]
Nov 02 07:47:23.893746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.893723    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-523de0-openshift-default-7659bd5f45-zx8hz"]
Nov 02 07:47:23.930668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.930632    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"]
Nov 02 07:47:23.934781 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:23.934759    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-54fd3c-openshift-default-694d595797-bw5dd"]
Nov 02 07:47:25.489416 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:25.489377    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" path="/var/lib/kubelet/pods/a5b71447-e15a-45e2-9230-4ec214621a65/volumes"
Nov 02 07:47:25.490175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:25.490153    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" path="/var/lib/kubelet/pods/b5f2b49f-9601-41b6-b9e2-5cbab0c8d226/volumes"
Nov 02 07:47:32.786480 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:32.786442    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"]
Nov 02 07:47:32.786935 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:32.786715    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy" containerID="cri-o://1003b2e05258003e59509a784df5ba85f90ebd1abde144d7c0a88db0ae171e2e" gracePeriod=30
Nov 02 07:47:38.115880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.115846    4898 generic.go:356] "Generic (PLEG): container finished" podID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerID="1003b2e05258003e59509a784df5ba85f90ebd1abde144d7c0a88db0ae171e2e" exitCode=0
Nov 02 07:47:38.116375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.115949    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" event={"ID":"2b859db2-8e95-4eab-8ebc-fb0e3d26b532","Type":"ContainerDied","Data":"1003b2e05258003e59509a784df5ba85f90ebd1abde144d7c0a88db0ae171e2e"}
Nov 02 07:47:38.116375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.116012    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l" event={"ID":"2b859db2-8e95-4eab-8ebc-fb0e3d26b532","Type":"ContainerDied","Data":"5dac18b4036cc5f8fb3321851e7932f4cb597d756d419de3aef19301c3104b8b"}
Nov 02 07:47:38.116375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.116032    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dac18b4036cc5f8fb3321851e7932f4cb597d756d419de3aef19301c3104b8b"
Nov 02 07:47:38.116375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.116049    4898 scope.go:122] "RemoveContainer" containerID="e5c1491d64ec98a8aa1b9e455614dd884ec260cf89591f035dac2943bc10ff98"
Nov 02 07:47:38.116375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.116346    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:47:38.239128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239096    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-credential-socket\") pod \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") "
Nov 02 07:47:38.239308 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239146    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-kube-api-access-l88hk\" (UniqueName: \"kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-kube-api-access-l88hk\") pod \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") "
Nov 02 07:47:38.239308 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239227    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-token\" (UniqueName: \"kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-token\") pod \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") "
Nov 02 07:47:38.239308 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239271    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-podinfo\") pod \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") "
Nov 02 07:47:38.239425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239391    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-data\") pod \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") "
Nov 02 07:47:38.239477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239427    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-credential-socket" pod "2b859db2-8e95-4eab-8ebc-fb0e3d26b532" (UID: "2b859db2-8e95-4eab-8ebc-fb0e3d26b532"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:38.239586 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239563    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-envoy\") pod \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") "
Nov 02 07:47:38.239657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239617    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-certs\") pod \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") "
Nov 02 07:47:38.239657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239633    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-data" pod "2b859db2-8e95-4eab-8ebc-fb0e3d26b532" (UID: "2b859db2-8e95-4eab-8ebc-fb0e3d26b532"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:38.239771 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239656    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-socket\") pod \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") "
Nov 02 07:47:38.239771 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239707    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istiod-ca-cert\") pod \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\" (UID: \"2b859db2-8e95-4eab-8ebc-fb0e3d26b532\") "
Nov 02 07:47:38.239968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239940    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-certs" pod "2b859db2-8e95-4eab-8ebc-fb0e3d26b532" (UID: "2b859db2-8e95-4eab-8ebc-fb0e3d26b532"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:38.239968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.239955    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-socket" pod "2b859db2-8e95-4eab-8ebc-fb0e3d26b532" (UID: "2b859db2-8e95-4eab-8ebc-fb0e3d26b532"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:38.240138 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.240114    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istiod-ca-cert" pod "2b859db2-8e95-4eab-8ebc-fb0e3d26b532" (UID: "2b859db2-8e95-4eab-8ebc-fb0e3d26b532"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:38.240448 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.240429    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:38.240546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.240454    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:38.240546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.240468    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:38.240546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.240480    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:38.240546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.240494    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:38.241723 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.241698    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-podinfo" pod "2b859db2-8e95-4eab-8ebc-fb0e3d26b532" (UID: "2b859db2-8e95-4eab-8ebc-fb0e3d26b532"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:47:38.241820 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.241703    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-envoy" pod "2b859db2-8e95-4eab-8ebc-fb0e3d26b532" (UID: "2b859db2-8e95-4eab-8ebc-fb0e3d26b532"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:47:38.241977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.241949    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-token" pod "2b859db2-8e95-4eab-8ebc-fb0e3d26b532" (UID: "2b859db2-8e95-4eab-8ebc-fb0e3d26b532"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:38.242142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.242123    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-kube-api-access-l88hk" pod "2b859db2-8e95-4eab-8ebc-fb0e3d26b532" (UID: "2b859db2-8e95-4eab-8ebc-fb0e3d26b532"). InnerVolumeSpecName "kube-api-access-l88hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:38.341633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.341556    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:38.341633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.341588    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l88hk\" (UniqueName: \"kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-kube-api-access-l88hk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:38.341633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.341600    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:38.341633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:38.341608    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b859db2-8e95-4eab-8ebc-fb0e3d26b532-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:39.125840 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:39.125807    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"
Nov 02 07:47:39.256030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:39.255995    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"]
Nov 02 07:47:39.267082 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:39.267046    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2cc1f-openshift-default-866f47f48c-hwn7l"]
Nov 02 07:47:39.485412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:39.485374    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" path="/var/lib/kubelet/pods/2b859db2-8e95-4eab-8ebc-fb0e3d26b532/volumes"
Nov 02 07:47:40.163158 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:40.163122    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"]
Nov 02 07:47:40.163664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:40.163363    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c" podUID="e99d42fa-2002-4206-8747-7a632e407bfa" containerName="payload-pre-processing" containerID="cri-o://cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd" gracePeriod=30
Nov 02 07:47:40.554087 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:40.553673    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"
Nov 02 07:47:40.733266 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:40.731237    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99d42fa-2002-4206-8747-7a632e407bfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d42fa-2002-4206-8747-7a632e407bfa-plugins-config-volume\") pod \"e99d42fa-2002-4206-8747-7a632e407bfa\" (UID: \"e99d42fa-2002-4206-8747-7a632e407bfa\") "
Nov 02 07:47:40.733807 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:40.733675    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99d42fa-2002-4206-8747-7a632e407bfa-kube-api-access-7bkxn\" (UniqueName: \"kubernetes.io/projected/e99d42fa-2002-4206-8747-7a632e407bfa-kube-api-access-7bkxn\") pod \"e99d42fa-2002-4206-8747-7a632e407bfa\" (UID: \"e99d42fa-2002-4206-8747-7a632e407bfa\") "
Nov 02 07:47:40.743819 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:40.741136    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99d42fa-2002-4206-8747-7a632e407bfa-plugins-config-volume" pod "e99d42fa-2002-4206-8747-7a632e407bfa" (UID: "e99d42fa-2002-4206-8747-7a632e407bfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:47:40.809103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:40.809027    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99d42fa-2002-4206-8747-7a632e407bfa-kube-api-access-7bkxn" pod "e99d42fa-2002-4206-8747-7a632e407bfa" (UID: "e99d42fa-2002-4206-8747-7a632e407bfa"). InnerVolumeSpecName "kube-api-access-7bkxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:47:40.835995 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:40.835959    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bkxn\" (UniqueName: \"kubernetes.io/projected/e99d42fa-2002-4206-8747-7a632e407bfa-kube-api-access-7bkxn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:40.835995 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:40.835987    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99d42fa-2002-4206-8747-7a632e407bfa-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:47:41.150187 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:41.150091    4898 generic.go:356] "Generic (PLEG): container finished" podID="e99d42fa-2002-4206-8747-7a632e407bfa" containerID="cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd" exitCode=0
Nov 02 07:47:41.150379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:41.150209    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c" event={"ID":"e99d42fa-2002-4206-8747-7a632e407bfa","Type":"ContainerDied","Data":"cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd"}
Nov 02 07:47:41.150379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:41.150269    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c" event={"ID":"e99d42fa-2002-4206-8747-7a632e407bfa","Type":"ContainerDied","Data":"67d030b2d3e73dad1d18d1128ccaa78019312e96de836e1a117c4022e59cfe52"}
Nov 02 07:47:41.150379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:41.150308    4898 scope.go:122] "RemoveContainer" containerID="cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd"
Nov 02 07:47:41.150564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:41.150430    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"
Nov 02 07:47:41.222098 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:41.215293    4898 scope.go:122] "RemoveContainer" containerID="cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd"
Nov 02 07:47:41.225931 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:47:41.225062    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd\": container with ID starting with cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd not found: ID does not exist" containerID="cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd"
Nov 02 07:47:41.225931 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:41.225114    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd"} err="failed to get container status \"cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd\": rpc error: code = NotFound desc = could not find container \"cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd\": container with ID starting with cec9795331d8a392f962fa8ad6183740be307eb2172b902b5d1755d425fedffd not found: ID does not exist"
Nov 02 07:47:41.369293 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:41.368954    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"]
Nov 02 07:47:41.391730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:41.390807    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cc3f7-56bc5549d7-l8v2c"]
Nov 02 07:47:41.468231 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:41.468186    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99d42fa-2002-4206-8747-7a632e407bfa" path="/var/lib/kubelet/pods/e99d42fa-2002-4206-8747-7a632e407bfa/volumes"
Nov 02 07:47:44.992650 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:47:44.992607    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:47:45.032840 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:45.032805    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"]
Nov 02 07:47:45.044763 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:45.044731    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d204bad-s27ql"]
Nov 02 07:47:45.467728 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:45.467691    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcea2473-5168-41f4-8d71-7e2136764f1b" path="/var/lib/kubelet/pods/bcea2473-5168-41f4-8d71-7e2136764f1b/volumes"
Nov 02 07:47:53.041927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:53.041871    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"]
Nov 02 07:47:53.051727 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:53.051694    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db66d052-cxrpn"]
Nov 02 07:47:53.466643 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:47:53.466611    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ad00cf-432f-4d56-add3-7330ca4bfa10" path="/var/lib/kubelet/pods/96ad00cf-432f-4d56-add3-7330ca4bfa10/volumes"
Nov 02 07:48:03.973911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:03.973850    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"]
Nov 02 07:48:03.974660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:03.974622    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy" containerID="cri-o://bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba" gracePeriod=30
Nov 02 07:48:08.175074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:08.175030    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"]
Nov 02 07:48:08.175603 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:08.175340    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" podUID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" containerName="istio-proxy" containerID="cri-o://46007fc9cf13af4ecf908433ea2442364f666219ff66dc5b674dea286731ab0c" gracePeriod=30
Nov 02 07:48:08.187513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:08.187482    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"]
Nov 02 07:48:08.187742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:08.187697    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" podUID="c4df2b59-c602-426a-b7f9-debe6dfce67c" containerName="istio-proxy" containerID="cri-o://f426e7fb0c5bab528775e3cc974786c2d687ef9e6af751ab6486cbc51aa71f98" gracePeriod=30
Nov 02 07:48:08.193500 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:08.193470    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"]
Nov 02 07:48:08.193738 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:08.193712    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5" podUID="a37f72a4-f404-47fd-9c2a-d5485fe59a49" containerName="istio-proxy" containerID="cri-o://936ebd104b41cc058dd75b209e9dff169f981a5dc11ac6cc40ea10f1fc7601bf" gracePeriod=30
Nov 02 07:48:09.234971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.234938    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:48:09.308262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.308179    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-envoy\") pod \"81861f8c-9af4-4530-b75a-6f8180c8f003\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") "
Nov 02 07:48:09.308438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.308279    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-certs\") pod \"81861f8c-9af4-4530-b75a-6f8180c8f003\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") "
Nov 02 07:48:09.308438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.308355    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81861f8c-9af4-4530-b75a-6f8180c8f003-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81861f8c-9af4-4530-b75a-6f8180c8f003-istiod-ca-cert\") pod \"81861f8c-9af4-4530-b75a-6f8180c8f003\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") "
Nov 02 07:48:09.308668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.308636    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-certs" pod "81861f8c-9af4-4530-b75a-6f8180c8f003" (UID: "81861f8c-9af4-4530-b75a-6f8180c8f003"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:09.308812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.308773    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81861f8c-9af4-4530-b75a-6f8180c8f003-istiod-ca-cert" pod "81861f8c-9af4-4530-b75a-6f8180c8f003" (UID: "81861f8c-9af4-4530-b75a-6f8180c8f003"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:09.309019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.308993    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-token\" (UniqueName: \"kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-token\") pod \"81861f8c-9af4-4530-b75a-6f8180c8f003\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") "
Nov 02 07:48:09.309158 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.309064    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-socket\") pod \"81861f8c-9af4-4530-b75a-6f8180c8f003\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") "
Nov 02 07:48:09.309158 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.309147    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-podinfo\") pod \"81861f8c-9af4-4530-b75a-6f8180c8f003\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") "
Nov 02 07:48:09.309271 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.309209    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-credential-socket\") pod \"81861f8c-9af4-4530-b75a-6f8180c8f003\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") "
Nov 02 07:48:09.309332 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.309270    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-data\") pod \"81861f8c-9af4-4530-b75a-6f8180c8f003\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") "
Nov 02 07:48:09.309416 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.309396    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-kube-api-access-zfdsl\" (UniqueName: \"kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-kube-api-access-zfdsl\") pod \"81861f8c-9af4-4530-b75a-6f8180c8f003\" (UID: \"81861f8c-9af4-4530-b75a-6f8180c8f003\") "
Nov 02 07:48:09.309851 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.309824    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-data" pod "81861f8c-9af4-4530-b75a-6f8180c8f003" (UID: "81861f8c-9af4-4530-b75a-6f8180c8f003"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:09.309952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.309842    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-credential-socket" pod "81861f8c-9af4-4530-b75a-6f8180c8f003" (UID: "81861f8c-9af4-4530-b75a-6f8180c8f003"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:09.309952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.309924    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-socket" pod "81861f8c-9af4-4530-b75a-6f8180c8f003" (UID: "81861f8c-9af4-4530-b75a-6f8180c8f003"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:09.310587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.310567    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.310704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.310690    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.310791 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.310778    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.310872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.310861    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.310994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.310978    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81861f8c-9af4-4530-b75a-6f8180c8f003-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.310994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.310955    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-envoy" pod "81861f8c-9af4-4530-b75a-6f8180c8f003" (UID: "81861f8c-9af4-4530-b75a-6f8180c8f003"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:09.312271 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.312246    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-podinfo" pod "81861f8c-9af4-4530-b75a-6f8180c8f003" (UID: "81861f8c-9af4-4530-b75a-6f8180c8f003"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:48:09.312488 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.312460    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-kube-api-access-zfdsl" pod "81861f8c-9af4-4530-b75a-6f8180c8f003" (UID: "81861f8c-9af4-4530-b75a-6f8180c8f003"). InnerVolumeSpecName "kube-api-access-zfdsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:09.314209 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.314186    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-token" pod "81861f8c-9af4-4530-b75a-6f8180c8f003" (UID: "81861f8c-9af4-4530-b75a-6f8180c8f003"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:09.412503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.412468    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.412503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.412505    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.412706 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.412522    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfdsl\" (UniqueName: \"kubernetes.io/projected/81861f8c-9af4-4530-b75a-6f8180c8f003-kube-api-access-zfdsl\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.412706 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.412537    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81861f8c-9af4-4530-b75a-6f8180c8f003-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:09.574722 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.574636    4898 generic.go:356] "Generic (PLEG): container finished" podID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerID="bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba" exitCode=0
Nov 02 07:48:09.574934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.574737    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"
Nov 02 07:48:09.574934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.574881    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" event={"ID":"81861f8c-9af4-4530-b75a-6f8180c8f003","Type":"ContainerDied","Data":"bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba"}
Nov 02 07:48:09.575074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.574951    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq" event={"ID":"81861f8c-9af4-4530-b75a-6f8180c8f003","Type":"ContainerDied","Data":"759715032d8706266b5c45c2b721dbc2f89fe8aa683397c5dcb727383e37b0a1"}
Nov 02 07:48:09.575074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.574984    4898 scope.go:122] "RemoveContainer" containerID="bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba"
Nov 02 07:48:09.586382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.586351    4898 scope.go:122] "RemoveContainer" containerID="b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a"
Nov 02 07:48:09.598788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.598765    4898 scope.go:122] "RemoveContainer" containerID="bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba"
Nov 02 07:48:09.599166 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:48:09.599139    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba\": container with ID starting with bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba not found: ID does not exist" containerID="bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba"
Nov 02 07:48:09.599270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.599176    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba"} err="failed to get container status \"bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba\": rpc error: code = NotFound desc = could not find container \"bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba\": container with ID starting with bb9d9bd4873f192a53a7f5f5c4e1c8d35a8043e7cfcc604c0cd567f75b97c5ba not found: ID does not exist"
Nov 02 07:48:09.599270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.599194    4898 scope.go:122] "RemoveContainer" containerID="b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a"
Nov 02 07:48:09.599495 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:48:09.599481    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a\": container with ID starting with b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a not found: ID does not exist" containerID="b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a"
Nov 02 07:48:09.599546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.599500    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a"} err="failed to get container status \"b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a\": rpc error: code = NotFound desc = could not find container \"b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a\": container with ID starting with b8cb9be129cd13d1bb2394444e9d3a697363080f61d64761dc6f92fac80dac2a not found: ID does not exist"
Nov 02 07:48:09.629832 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.629801    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"]
Nov 02 07:48:09.642271 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:09.642242    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-dc62c2-openshift-default-995df4cd9-ccpjq"]
Nov 02 07:48:10.971509 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:10.971471    4898 scope.go:122] "RemoveContainer" containerID="64c2d9d6cba14a333927931e5857d5a56d74629021b098c07aeda98510af10ea"
Nov 02 07:48:11.489975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:11.489936    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" path="/var/lib/kubelet/pods/81861f8c-9af4-4530-b75a-6f8180c8f003/volumes"
Nov 02 07:48:13.618910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.618392    4898 generic.go:356] "Generic (PLEG): container finished" podID="c4df2b59-c602-426a-b7f9-debe6dfce67c" containerID="f426e7fb0c5bab528775e3cc974786c2d687ef9e6af751ab6486cbc51aa71f98" exitCode=0
Nov 02 07:48:13.618910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.618592    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" event={"ID":"c4df2b59-c602-426a-b7f9-debe6dfce67c","Type":"ContainerDied","Data":"f426e7fb0c5bab528775e3cc974786c2d687ef9e6af751ab6486cbc51aa71f98"}
Nov 02 07:48:13.624205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.620991    4898 generic.go:356] "Generic (PLEG): container finished" podID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" containerID="46007fc9cf13af4ecf908433ea2442364f666219ff66dc5b674dea286731ab0c" exitCode=0
Nov 02 07:48:13.624205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.621144    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" event={"ID":"cb10a74e-3c67-4331-9cc8-c54ce4a948ba","Type":"ContainerDied","Data":"46007fc9cf13af4ecf908433ea2442364f666219ff66dc5b674dea286731ab0c"}
Nov 02 07:48:13.624205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.623919    4898 generic.go:356] "Generic (PLEG): container finished" podID="a37f72a4-f404-47fd-9c2a-d5485fe59a49" containerID="936ebd104b41cc058dd75b209e9dff169f981a5dc11ac6cc40ea10f1fc7601bf" exitCode=0
Nov 02 07:48:13.624205 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.624005    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5" event={"ID":"a37f72a4-f404-47fd-9c2a-d5485fe59a49","Type":"ContainerDied","Data":"936ebd104b41cc058dd75b209e9dff169f981a5dc11ac6cc40ea10f1fc7601bf"}
Nov 02 07:48:13.710650 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.710617    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:48:13.731982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.731935    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-envoy\") pod \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") "
Nov 02 07:48:13.732242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.732214    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-certs\") pod \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") "
Nov 02 07:48:13.732406 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.732325    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-kube-api-access-rqt4r\" (UniqueName: \"kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-kube-api-access-rqt4r\") pod \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") "
Nov 02 07:48:13.732406 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.732386    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-credential-socket\") pod \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") "
Nov 02 07:48:13.732621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.732420    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-data\") pod \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") "
Nov 02 07:48:13.732621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.732468    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-socket\") pod \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") "
Nov 02 07:48:13.732752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.732653    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-token\" (UniqueName: \"kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-token\") pod \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") "
Nov 02 07:48:13.732752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.732700    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istiod-ca-cert\") pod \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") "
Nov 02 07:48:13.732752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.732731    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-podinfo\") pod \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\" (UID: \"cb10a74e-3c67-4331-9cc8-c54ce4a948ba\") "
Nov 02 07:48:13.734225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.733999    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-data" pod "cb10a74e-3c67-4331-9cc8-c54ce4a948ba" (UID: "cb10a74e-3c67-4331-9cc8-c54ce4a948ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.734696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.734477    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-certs" pod "cb10a74e-3c67-4331-9cc8-c54ce4a948ba" (UID: "cb10a74e-3c67-4331-9cc8-c54ce4a948ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.734853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.734797    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-socket" pod "cb10a74e-3c67-4331-9cc8-c54ce4a948ba" (UID: "cb10a74e-3c67-4331-9cc8-c54ce4a948ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.734853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.734811    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-credential-socket" pod "cb10a74e-3c67-4331-9cc8-c54ce4a948ba" (UID: "cb10a74e-3c67-4331-9cc8-c54ce4a948ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.735174 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.735152    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istiod-ca-cert" pod "cb10a74e-3c67-4331-9cc8-c54ce4a948ba" (UID: "cb10a74e-3c67-4331-9cc8-c54ce4a948ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:13.736484 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.736437    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-envoy" pod "cb10a74e-3c67-4331-9cc8-c54ce4a948ba" (UID: "cb10a74e-3c67-4331-9cc8-c54ce4a948ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.737494 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.737460    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-token" pod "cb10a74e-3c67-4331-9cc8-c54ce4a948ba" (UID: "cb10a74e-3c67-4331-9cc8-c54ce4a948ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:13.738624 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.738602    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-kube-api-access-rqt4r" pod "cb10a74e-3c67-4331-9cc8-c54ce4a948ba" (UID: "cb10a74e-3c67-4331-9cc8-c54ce4a948ba"). InnerVolumeSpecName "kube-api-access-rqt4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:13.738717 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.738665    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-podinfo" pod "cb10a74e-3c67-4331-9cc8-c54ce4a948ba" (UID: "cb10a74e-3c67-4331-9cc8-c54ce4a948ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:48:13.765427 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.765334    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:48:13.768276 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.768119    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:48:13.834335 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834297    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-data\") pod \"c4df2b59-c602-426a-b7f9-debe6dfce67c\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") "
Nov 02 07:48:13.834537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834358    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-socket\") pod \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") "
Nov 02 07:48:13.834537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834416    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-certs\") pod \"c4df2b59-c602-426a-b7f9-debe6dfce67c\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") "
Nov 02 07:48:13.834537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834447    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-token\" (UniqueName: \"kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-token\") pod \"c4df2b59-c602-426a-b7f9-debe6dfce67c\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") "
Nov 02 07:48:13.834537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834474    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4df2b59-c602-426a-b7f9-debe6dfce67c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4df2b59-c602-426a-b7f9-debe6dfce67c-istiod-ca-cert\") pod \"c4df2b59-c602-426a-b7f9-debe6dfce67c\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") "
Nov 02 07:48:13.834537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834499    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-credential-socket\") pod \"c4df2b59-c602-426a-b7f9-debe6dfce67c\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") "
Nov 02 07:48:13.834822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834537    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istiod-ca-cert\") pod \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") "
Nov 02 07:48:13.834822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834598    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-podinfo\") pod \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") "
Nov 02 07:48:13.834822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834649    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-data\") pod \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") "
Nov 02 07:48:13.834822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834628    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-data" pod "c4df2b59-c602-426a-b7f9-debe6dfce67c" (UID: "c4df2b59-c602-426a-b7f9-debe6dfce67c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.834822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834699    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-socket" pod "a37f72a4-f404-47fd-9c2a-d5485fe59a49" (UID: "a37f72a4-f404-47fd-9c2a-d5485fe59a49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.834822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834759    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-kube-api-access-cjfgp\" (UniqueName: \"kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-kube-api-access-cjfgp\") pod \"c4df2b59-c602-426a-b7f9-debe6dfce67c\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") "
Nov 02 07:48:13.834822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834800    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-envoy\") pod \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") "
Nov 02 07:48:13.835211 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834829    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-socket\") pod \"c4df2b59-c602-426a-b7f9-debe6dfce67c\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") "
Nov 02 07:48:13.835211 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834934    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-credential-socket\") pod \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") "
Nov 02 07:48:13.835211 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.834996    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-certs\") pod \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") "
Nov 02 07:48:13.835211 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.835035    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4df2b59-c602-426a-b7f9-debe6dfce67c-istiod-ca-cert" pod "c4df2b59-c602-426a-b7f9-debe6dfce67c" (UID: "c4df2b59-c602-426a-b7f9-debe6dfce67c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:13.835211 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.835145    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istiod-ca-cert" pod "a37f72a4-f404-47fd-9c2a-d5485fe59a49" (UID: "a37f72a4-f404-47fd-9c2a-d5485fe59a49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:13.835211 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.835115    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-credential-socket" pod "c4df2b59-c602-426a-b7f9-debe6dfce67c" (UID: "c4df2b59-c602-426a-b7f9-debe6dfce67c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.835211 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.835107    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-kube-api-access-5swcn\" (UniqueName: \"kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-kube-api-access-5swcn\") pod \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") "
Nov 02 07:48:13.835711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.835285    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-podinfo\") pod \"c4df2b59-c602-426a-b7f9-debe6dfce67c\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") "
Nov 02 07:48:13.835711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.835292    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-certs" pod "c4df2b59-c602-426a-b7f9-debe6dfce67c" (UID: "c4df2b59-c602-426a-b7f9-debe6dfce67c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.835711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.835358    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-token\" (UniqueName: \"kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-token\") pod \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\" (UID: \"a37f72a4-f404-47fd-9c2a-d5485fe59a49\") "
Nov 02 07:48:13.835711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.835468    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-envoy\") pod \"c4df2b59-c602-426a-b7f9-debe6dfce67c\" (UID: \"c4df2b59-c602-426a-b7f9-debe6dfce67c\") "
Nov 02 07:48:13.835711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.835487    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-credential-socket" pod "a37f72a4-f404-47fd-9c2a-d5485fe59a49" (UID: "a37f72a4-f404-47fd-9c2a-d5485fe59a49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.835711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.835668    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-data" pod "a37f72a4-f404-47fd-9c2a-d5485fe59a49" (UID: "a37f72a4-f404-47fd-9c2a-d5485fe59a49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.835711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.835696    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-socket" pod "c4df2b59-c602-426a-b7f9-debe6dfce67c" (UID: "c4df2b59-c602-426a-b7f9-debe6dfce67c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836194    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-certs" pod "a37f72a4-f404-47fd-9c2a-d5485fe59a49" (UID: "a37f72a4-f404-47fd-9c2a-d5485fe59a49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836303    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqt4r\" (UniqueName: \"kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-kube-api-access-rqt4r\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836320    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836330    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836339    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836348    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836357    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836366    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836375    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4df2b59-c602-426a-b7f9-debe6dfce67c-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836382    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836392    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836400    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836408    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836415    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836423    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836430    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.836425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836438    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.837195 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836446    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb10a74e-3c67-4331-9cc8-c54ce4a948ba-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.837195 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.836456    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.839958 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.839874    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-token" pod "c4df2b59-c602-426a-b7f9-debe6dfce67c" (UID: "c4df2b59-c602-426a-b7f9-debe6dfce67c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:13.840260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.840234    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-podinfo" pod "a37f72a4-f404-47fd-9c2a-d5485fe59a49" (UID: "a37f72a4-f404-47fd-9c2a-d5485fe59a49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:48:13.840360 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.840319    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-podinfo" pod "c4df2b59-c602-426a-b7f9-debe6dfce67c" (UID: "c4df2b59-c602-426a-b7f9-debe6dfce67c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:48:13.840512 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.840491    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-envoy" pod "a37f72a4-f404-47fd-9c2a-d5485fe59a49" (UID: "a37f72a4-f404-47fd-9c2a-d5485fe59a49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.840590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.840566    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-token" pod "a37f72a4-f404-47fd-9c2a-d5485fe59a49" (UID: "a37f72a4-f404-47fd-9c2a-d5485fe59a49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:13.840655 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.840637    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-kube-api-access-5swcn" pod "a37f72a4-f404-47fd-9c2a-d5485fe59a49" (UID: "a37f72a4-f404-47fd-9c2a-d5485fe59a49"). InnerVolumeSpecName "kube-api-access-5swcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:13.841651 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.841629    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-envoy" pod "c4df2b59-c602-426a-b7f9-debe6dfce67c" (UID: "c4df2b59-c602-426a-b7f9-debe6dfce67c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:13.842409 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.842389    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-kube-api-access-cjfgp" pod "c4df2b59-c602-426a-b7f9-debe6dfce67c" (UID: "c4df2b59-c602-426a-b7f9-debe6dfce67c"). InnerVolumeSpecName "kube-api-access-cjfgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:13.938191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.938155    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjfgp\" (UniqueName: \"kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-kube-api-access-cjfgp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.938191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.938182    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.938191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.938191    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37f72a4-f404-47fd-9c2a-d5485fe59a49-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.938191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.938201    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5swcn\" (UniqueName: \"kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-kube-api-access-5swcn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.938543 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.938210    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.938543 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.938219    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.938543 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.938227    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.938543 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.938236    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4df2b59-c602-426a-b7f9-debe6dfce67c-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:13.938543 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:13.938244    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37f72a4-f404-47fd-9c2a-d5485fe59a49-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:14.632984 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.632863    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"
Nov 02 07:48:14.632984 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.632873    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5" event={"ID":"a37f72a4-f404-47fd-9c2a-d5485fe59a49","Type":"ContainerDied","Data":"62bba73e32328f887ba39d8bb57d380a6644eed8751ed2e5d06afde0fbc5da21"}
Nov 02 07:48:14.632984 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.632983    4898 scope.go:122] "RemoveContainer" containerID="936ebd104b41cc058dd75b209e9dff169f981a5dc11ac6cc40ea10f1fc7601bf"
Nov 02 07:48:14.634464 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.634440    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"
Nov 02 07:48:14.634590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.634474    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt" event={"ID":"c4df2b59-c602-426a-b7f9-debe6dfce67c","Type":"ContainerDied","Data":"e447ae09a5e1abd46e5e405201997c26f8023262ca822eee68e7bc7717fb5994"}
Nov 02 07:48:14.636249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.636221    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54" event={"ID":"cb10a74e-3c67-4331-9cc8-c54ce4a948ba","Type":"ContainerDied","Data":"dd750b3c638b8d6432bf5399fd2b656319fe3c0b1f45a3b1f7f00bcd56181a5c"}
Nov 02 07:48:14.636351 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.636272    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"
Nov 02 07:48:14.647767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.647746    4898 scope.go:122] "RemoveContainer" containerID="f426e7fb0c5bab528775e3cc974786c2d687ef9e6af751ab6486cbc51aa71f98"
Nov 02 07:48:14.658976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.658953    4898 scope.go:122] "RemoveContainer" containerID="46007fc9cf13af4ecf908433ea2442364f666219ff66dc5b674dea286731ab0c"
Nov 02 07:48:14.700178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.700139    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"]
Nov 02 07:48:14.712183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.712146    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-kf9wt"]
Nov 02 07:48:14.738775 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.738738    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"]
Nov 02 07:48:14.742664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.742637    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-bhdc5"]
Nov 02 07:48:14.780444 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.780412    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"]
Nov 02 07:48:14.789234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:14.789207    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zvg54"]
Nov 02 07:48:14.996034 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:48:14.995988    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:48:15.468024 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:15.467987    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37f72a4-f404-47fd-9c2a-d5485fe59a49" path="/var/lib/kubelet/pods/a37f72a4-f404-47fd-9c2a-d5485fe59a49/volumes"
Nov 02 07:48:15.468610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:15.468586    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4df2b59-c602-426a-b7f9-debe6dfce67c" path="/var/lib/kubelet/pods/c4df2b59-c602-426a-b7f9-debe6dfce67c/volumes"
Nov 02 07:48:15.469153 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:15.469133    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" path="/var/lib/kubelet/pods/cb10a74e-3c67-4331-9cc8-c54ce4a948ba/volumes"
Nov 02 07:48:20.031346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:20.031309    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"]
Nov 02 07:48:20.043155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:20.043122    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f8ece77-nlmzh"]
Nov 02 07:48:21.466526 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:21.466493    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e01286-cce2-4704-b024-cb73980f5d9a" path="/var/lib/kubelet/pods/b4e01286-cce2-4704-b024-cb73980f5d9a/volumes"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.757349    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"]
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758336    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758360    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758381    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758390    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758403    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758410    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758420    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758427    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758441    4898 cpu_manager.go:414] "containerMap: removing container" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758449    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758481    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758490    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758517    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c4df2b59-c602-426a-b7f9-debe6dfce67c" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758522    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4df2b59-c602-426a-b7f9-debe6dfce67c" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758528    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758533    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758550    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6ff0ab99-f59f-4b01-a2f5-704b760da63f" containerName="payload-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758554    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff0ab99-f59f-4b01-a2f5-704b760da63f" containerName="payload-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758564    4898 cpu_manager.go:414] "containerMap: removing container" podUID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758570    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758579    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758584    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758596    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2be61eaf-5961-4f71-a125-037d47ac2c54" containerName="payload-pre-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758601    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be61eaf-5961-4f71-a125-037d47ac2c54" containerName="payload-pre-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758609    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a37f72a4-f404-47fd-9c2a-d5485fe59a49" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758613    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37f72a4-f404-47fd-9c2a-d5485fe59a49" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758628    4898 cpu_manager.go:414] "containerMap: removing container" podUID="07f3f205-27d6-4401-be74-e655ca60c83a" containerName="payload-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758632    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f3f205-27d6-4401-be74-e655ca60c83a" containerName="payload-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758640    4898 cpu_manager.go:414] "containerMap: removing container" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758644    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758650    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e99d42fa-2002-4206-8747-7a632e407bfa" containerName="payload-pre-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758655    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99d42fa-2002-4206-8747-7a632e407bfa" containerName="payload-pre-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758665    4898 cpu_manager.go:414] "containerMap: removing container" podUID="898cbf78-6478-41a2-937d-30f8e5da22f2" containerName="payload-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758670    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="898cbf78-6478-41a2-937d-30f8e5da22f2" containerName="payload-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758676    4898 cpu_manager.go:414] "containerMap: removing container" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758680    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758690    4898 cpu_manager.go:414] "containerMap: removing container" podUID="262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1" containerName="payload-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758694    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1" containerName="payload-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758934    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4df2b59-c602-426a-b7f9-debe6dfce67c" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758951    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758963    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff0ab99-f59f-4b01-a2f5-704b760da63f" containerName="payload-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758973    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb10a74e-3c67-4331-9cc8-c54ce4a948ba" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758984    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b859db2-8e95-4eab-8ebc-fb0e3d26b532" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.758994    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be61eaf-5961-4f71-a125-037d47ac2c54" containerName="payload-pre-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759004    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37f72a4-f404-47fd-9c2a-d5485fe59a49" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759010    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759018    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f3f205-27d6-4401-be74-e655ca60c83a" containerName="payload-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759024    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759032    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99d42fa-2002-4206-8747-7a632e407bfa" containerName="payload-pre-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759038    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="898cbf78-6478-41a2-937d-30f8e5da22f2" containerName="payload-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759043    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fbf728-4e49-4574-9bc8-1d53e3356957" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759049    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="262cad8c-e649-4ea8-b7b0-d2e5cd4ee4f1" containerName="payload-processing"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759056    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759062    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759068    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f85612-ef12-4dad-826f-3711b7f72703" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759075    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b71447-e15a-45e2-9230-4ec214621a65" containerName="istio-proxy"
Nov 02 07:48:24.759976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.759081    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f2b49f-9601-41b6-b9e2-5cbab0c8d226" containerName="istio-proxy"
Nov 02 07:48:24.764014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.763990    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"
Nov 02 07:48:24.809878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.804160    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9f3716\""
Nov 02 07:48:24.809878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.804933    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9f3716-dockercfg-d4vlh\""
Nov 02 07:48:24.831794 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.831762    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"]
Nov 02 07:48:24.948214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.945318    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jm9\" (UniqueName: \"kubernetes.io/projected/7ccd53d3-0f42-423c-82bd-642e4de884a3-kube-api-access-c9jm9\") pod \"payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv\" (UID: \"7ccd53d3-0f42-423c-82bd-642e4de884a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"
Nov 02 07:48:24.948214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:24.945410    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ccd53d3-0f42-423c-82bd-642e4de884a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv\" (UID: \"7ccd53d3-0f42-423c-82bd-642e4de884a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"
Nov 02 07:48:25.046507 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.046420    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jm9\" (UniqueName: \"kubernetes.io/projected/7ccd53d3-0f42-423c-82bd-642e4de884a3-kube-api-access-c9jm9\") pod \"payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv\" (UID: \"7ccd53d3-0f42-423c-82bd-642e4de884a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"
Nov 02 07:48:25.046507 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.046496    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ccd53d3-0f42-423c-82bd-642e4de884a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv\" (UID: \"7ccd53d3-0f42-423c-82bd-642e4de884a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"
Nov 02 07:48:25.047236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.047215    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ccd53d3-0f42-423c-82bd-642e4de884a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv\" (UID: \"7ccd53d3-0f42-423c-82bd-642e4de884a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"
Nov 02 07:48:25.069957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.069920    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jm9\" (UniqueName: \"kubernetes.io/projected/7ccd53d3-0f42-423c-82bd-642e4de884a3-kube-api-access-c9jm9\") pod \"payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv\" (UID: \"7ccd53d3-0f42-423c-82bd-642e4de884a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"
Nov 02 07:48:25.130754 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.130716    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"
Nov 02 07:48:25.509621 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:48:25.509585    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccd53d3_0f42_423c_82bd_642e4de884a3.slice/crio-41a4e5d854dd66c8591a24a4d698f346a34ebfcf53627cbd9e7a92928c675d53 WatchSource:0}: Error finding container 41a4e5d854dd66c8591a24a4d698f346a34ebfcf53627cbd9e7a92928c675d53: Status 404 returned error can't find the container with id 41a4e5d854dd66c8591a24a4d698f346a34ebfcf53627cbd9e7a92928c675d53
Nov 02 07:48:25.512321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.512293    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"]
Nov 02 07:48:25.726289 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.726254    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv" event={"ID":"7ccd53d3-0f42-423c-82bd-642e4de884a3","Type":"ContainerStarted","Data":"75267ce18581210c00cd48bfb40cc4f391b687bed55bb86c9f79a391e50390c4"}
Nov 02 07:48:25.726498 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.726313    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv" event={"ID":"7ccd53d3-0f42-423c-82bd-642e4de884a3","Type":"ContainerStarted","Data":"41a4e5d854dd66c8591a24a4d698f346a34ebfcf53627cbd9e7a92928c675d53"}
Nov 02 07:48:25.726498 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.726342    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"
Nov 02 07:48:25.768841 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.768719    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv" podStartSLOduration=1.768701132 podStartE2EDuration="1.768701132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:25.767001481 +0000 UTC m=+1456.457328445" watchObservedRunningTime="2026-11-02 07:48:25.768701132 +0000 UTC m=+1456.459028098"
Nov 02 07:48:25.795742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.795710    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"]
Nov 02 07:48:25.797855 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.796467    4898 cpu_manager.go:414] "containerMap: removing container" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy"
Nov 02 07:48:25.797855 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.796486    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy"
Nov 02 07:48:25.797855 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.796629    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="81861f8c-9af4-4530-b75a-6f8180c8f003" containerName="istio-proxy"
Nov 02 07:48:25.801309 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.800728    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.822918 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.821922    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"]
Nov 02 07:48:25.861379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.861329    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-certs\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.861621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.861391    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-credential-socket\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.861621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.861425    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-socket\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.861621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.861450    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-token\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.861621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.861507    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvrnk\" (UniqueName: \"kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-kube-api-access-pvrnk\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.861848 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.861704    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istiod-ca-cert\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.861848 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.861807    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-envoy\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.862059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.861880    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-podinfo\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.862059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.861935    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-data\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.863709 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.863685    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b2f76f-openshift-default-dockercfg-7svsz\""
Nov 02 07:48:25.964280 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.964239    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istiod-ca-cert\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.964474 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.964335    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-envoy\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.964474 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.964424    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-podinfo\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.964474 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.964467    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-data\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.964613 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.964554    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-certs\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.964613 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.964584    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-credential-socket\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.964699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.964615    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-socket\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.964699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.964645    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-token\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.964778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.964712    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvrnk\" (UniqueName: \"kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-kube-api-access-pvrnk\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.965870 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.965842    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istiod-ca-cert\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.966385 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.966300    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-certs\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.966385 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.966346    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-socket\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.966566 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.966448    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-credential-socket\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.966641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.966621    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-data\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.968813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.968780    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-envoy\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.968982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.968847    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-podinfo\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.989630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.989586    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvrnk\" (UniqueName: \"kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-kube-api-access-pvrnk\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:25.989920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:25.989880    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-token\") pod \"e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") " pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:26.135503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:26.135157    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:26.615225 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:48:26.615188    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c749a89_2eb7_4bdf_810a_ffae61b7e264.slice/crio-1baa1af1ab1651f3231b8d25f95be05150c3bbf787b389f728ed12ec639fd279 WatchSource:0}: Error finding container 1baa1af1ab1651f3231b8d25f95be05150c3bbf787b389f728ed12ec639fd279: Status 404 returned error can't find the container with id 1baa1af1ab1651f3231b8d25f95be05150c3bbf787b389f728ed12ec639fd279
Nov 02 07:48:26.615649 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:26.615621    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"]
Nov 02 07:48:26.618010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:26.617975    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:26.618115 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:26.618055    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:26.618115 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:26.618100    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:26.784281 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:26.784195    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" event={"ID":"5c749a89-2eb7-4bdf-810a-ffae61b7e264","Type":"ContainerStarted","Data":"1baa1af1ab1651f3231b8d25f95be05150c3bbf787b389f728ed12ec639fd279"}
Nov 02 07:48:27.793962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:27.793927    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" event={"ID":"5c749a89-2eb7-4bdf-810a-ffae61b7e264","Type":"ContainerStarted","Data":"612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea"}
Nov 02 07:48:27.899825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:27.899768    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" podStartSLOduration=2.899753086 podStartE2EDuration="2.899753086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:27.896176031 +0000 UTC m=+1458.586502996" watchObservedRunningTime="2026-11-02 07:48:27.899753086 +0000 UTC m=+1458.590080101"
Nov 02 07:48:28.136154 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:28.136056    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:28.142117 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:28.142088    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:28.801571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:28.801535    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:28.802521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:28.802497    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:48:35.025438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:35.025399    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"]
Nov 02 07:48:35.026145 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:35.025623    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy" containerID="cri-o://f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879" gracePeriod=30
Nov 02 07:48:35.039245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:35.039166    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"]
Nov 02 07:48:35.039490 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:35.039463    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g" podUID="1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" containerName="istio-proxy" containerID="cri-o://073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071" gracePeriod=30
Nov 02 07:48:35.043650 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:35.043625    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"]
Nov 02 07:48:35.043827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:35.043807    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79" podUID="64107366-d1ff-46e2-8883-f21ae646ef25" containerName="istio-proxy" containerID="cri-o://4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3" gracePeriod=30
Nov 02 07:48:36.790832 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:36.790380    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"
Nov 02 07:48:40.378731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.378695    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:48:40.412447 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.411808    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:48:40.453694 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.453657    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:48:40.467305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467271    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-data\") pod \"64107366-d1ff-46e2-8883-f21ae646ef25\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") "
Nov 02 07:48:40.467480 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467316    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istiod-ca-cert\") pod \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") "
Nov 02 07:48:40.467480 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467351    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-credential-socket\") pod \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") "
Nov 02 07:48:40.467480 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467395    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-istio-token\" (UniqueName: \"kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-istio-token\") pod \"64107366-d1ff-46e2-8883-f21ae646ef25\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467593    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-data" pod "64107366-d1ff-46e2-8883-f21ae646ef25" (UID: "64107366-d1ff-46e2-8883-f21ae646ef25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467656    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-credential-socket" pod "06eb45ea-2696-40be-9e1f-90bc0fe6fead" (UID: "06eb45ea-2696-40be-9e1f-90bc0fe6fead"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467709    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-certs\") pod \"64107366-d1ff-46e2-8883-f21ae646ef25\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467793    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istiod-ca-cert" pod "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" (UID: "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467846    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-envoy\") pod \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467907    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-socket\") pod \"64107366-d1ff-46e2-8883-f21ae646ef25\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467938    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-credential-socket\") pod \"64107366-d1ff-46e2-8883-f21ae646ef25\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467970    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-podinfo\") pod \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.467967    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-certs" pod "64107366-d1ff-46e2-8883-f21ae646ef25" (UID: "64107366-d1ff-46e2-8883-f21ae646ef25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468000    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-envoy\") pod \"64107366-d1ff-46e2-8883-f21ae646ef25\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468043    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-certs\") pod \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468073    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64107366-d1ff-46e2-8883-f21ae646ef25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64107366-d1ff-46e2-8883-f21ae646ef25-istio-podinfo\") pod \"64107366-d1ff-46e2-8883-f21ae646ef25\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468101    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-data\") pod \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468125    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-socket\") pod \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468169    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-socket" pod "64107366-d1ff-46e2-8883-f21ae646ef25" (UID: "64107366-d1ff-46e2-8883-f21ae646ef25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468180    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istiod-ca-cert\") pod \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468229    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64107366-d1ff-46e2-8883-f21ae646ef25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64107366-d1ff-46e2-8883-f21ae646ef25-istiod-ca-cert\") pod \"64107366-d1ff-46e2-8883-f21ae646ef25\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468289    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-kube-api-access-56rdm\" (UniqueName: \"kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-kube-api-access-56rdm\") pod \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468347    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-data\") pod \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468397    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-credential-socket\") pod \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468445    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-podinfo\") pod \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468490    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-token\" (UniqueName: \"kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-token\") pod \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468519    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-token\" (UniqueName: \"kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-token\") pod \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468558    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-envoy\") pod \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\" (UID: \"06eb45ea-2696-40be-9e1f-90bc0fe6fead\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468595    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-certs\") pod \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468631    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-kube-api-access-jkxv4\" (UniqueName: \"kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-kube-api-access-jkxv4\") pod \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468745    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-socket\") pod \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\" (UID: \"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.468774    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-kube-api-access-54rj7\" (UniqueName: \"kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-kube-api-access-54rj7\") pod \"64107366-d1ff-46e2-8883-f21ae646ef25\" (UID: \"64107366-d1ff-46e2-8883-f21ae646ef25\") "
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.469395    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-certs" pod "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" (UID: "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.469537    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.469552    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.469562    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.469570    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.469580    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.469590    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.469820    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-data" pod "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" (UID: "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.470023    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-certs" pod "06eb45ea-2696-40be-9e1f-90bc0fe6fead" (UID: "06eb45ea-2696-40be-9e1f-90bc0fe6fead"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.470159    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-credential-socket" pod "64107366-d1ff-46e2-8883-f21ae646ef25" (UID: "64107366-d1ff-46e2-8883-f21ae646ef25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.470457    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64107366-d1ff-46e2-8883-f21ae646ef25-istiod-ca-cert" pod "64107366-d1ff-46e2-8883-f21ae646ef25" (UID: "64107366-d1ff-46e2-8883-f21ae646ef25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:40.471124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.470817    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istiod-ca-cert" pod "06eb45ea-2696-40be-9e1f-90bc0fe6fead" (UID: "06eb45ea-2696-40be-9e1f-90bc0fe6fead"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:48:40.473610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.471549    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-istio-token" pod "64107366-d1ff-46e2-8883-f21ae646ef25" (UID: "64107366-d1ff-46e2-8883-f21ae646ef25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:40.473610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.471769    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-data" pod "06eb45ea-2696-40be-9e1f-90bc0fe6fead" (UID: "06eb45ea-2696-40be-9e1f-90bc0fe6fead"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.474250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.474193    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-envoy" pod "06eb45ea-2696-40be-9e1f-90bc0fe6fead" (UID: "06eb45ea-2696-40be-9e1f-90bc0fe6fead"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.475150 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.475122    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-podinfo" pod "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" (UID: "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:48:40.475380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.475356    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-credential-socket" pod "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" (UID: "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.475573 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.475551    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-socket" pod "06eb45ea-2696-40be-9e1f-90bc0fe6fead" (UID: "06eb45ea-2696-40be-9e1f-90bc0fe6fead"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.476254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.476225    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-kube-api-access-54rj7" pod "64107366-d1ff-46e2-8883-f21ae646ef25" (UID: "64107366-d1ff-46e2-8883-f21ae646ef25"). InnerVolumeSpecName "kube-api-access-54rj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:40.476538 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.476509    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-token" pod "06eb45ea-2696-40be-9e1f-90bc0fe6fead" (UID: "06eb45ea-2696-40be-9e1f-90bc0fe6fead"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:40.476652 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.476589    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-socket" pod "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" (UID: "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.478087 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.478056    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64107366-d1ff-46e2-8883-f21ae646ef25-istio-podinfo" pod "64107366-d1ff-46e2-8883-f21ae646ef25" (UID: "64107366-d1ff-46e2-8883-f21ae646ef25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:48:40.478192 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.478123    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-kube-api-access-jkxv4" pod "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" (UID: "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e"). InnerVolumeSpecName "kube-api-access-jkxv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:40.478451 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.478428    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-envoy" pod "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" (UID: "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.479327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.479301    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-kube-api-access-56rdm" pod "06eb45ea-2696-40be-9e1f-90bc0fe6fead" (UID: "06eb45ea-2696-40be-9e1f-90bc0fe6fead"). InnerVolumeSpecName "kube-api-access-56rdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:40.479431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.479323    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-token" pod "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" (UID: "1d4e4ee9-0e4d-478f-9a88-8dc4c082150e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:48:40.480848 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.480821    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-podinfo" pod "06eb45ea-2696-40be-9e1f-90bc0fe6fead" (UID: "06eb45ea-2696-40be-9e1f-90bc0fe6fead"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:48:40.481468 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.481430    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-envoy" pod "64107366-d1ff-46e2-8883-f21ae646ef25" (UID: "64107366-d1ff-46e2-8883-f21ae646ef25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:48:40.571400 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571360    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571400 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571400    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54rj7\" (UniqueName: \"kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-kube-api-access-54rj7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571418    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64107366-d1ff-46e2-8883-f21ae646ef25-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571433    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571447    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571461    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571473    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64107366-d1ff-46e2-8883-f21ae646ef25-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571486    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571498    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64107366-d1ff-46e2-8883-f21ae646ef25-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571511    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571526    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571539    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571551    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64107366-d1ff-46e2-8883-f21ae646ef25-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571562    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56rdm\" (UniqueName: \"kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-kube-api-access-56rdm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571574    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571589    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571603    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.571611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571615    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.572157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571629    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.572157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571643    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06eb45ea-2696-40be-9e1f-90bc0fe6fead-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.572157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.571655    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkxv4\" (UniqueName: \"kubernetes.io/projected/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e-kube-api-access-jkxv4\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:48:40.916340 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.916294    4898 generic.go:356] "Generic (PLEG): container finished" podID="64107366-d1ff-46e2-8883-f21ae646ef25" containerID="4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3" exitCode=0
Nov 02 07:48:40.916340 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.916332    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79" event={"ID":"64107366-d1ff-46e2-8883-f21ae646ef25","Type":"ContainerDied","Data":"4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3"}
Nov 02 07:48:40.916681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.916371    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"
Nov 02 07:48:40.916681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.916435    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79" event={"ID":"64107366-d1ff-46e2-8883-f21ae646ef25","Type":"ContainerDied","Data":"7773ffac4b870af8e5803036a1ef1d541cf4804213305edf95298cde6599c984"}
Nov 02 07:48:40.916681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.916464    4898 scope.go:122] "RemoveContainer" containerID="4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3"
Nov 02 07:48:40.918704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.918287    4898 generic.go:356] "Generic (PLEG): container finished" podID="1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" containerID="073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071" exitCode=0
Nov 02 07:48:40.918704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.918412    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"
Nov 02 07:48:40.918704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.918457    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g" event={"ID":"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e","Type":"ContainerDied","Data":"073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071"}
Nov 02 07:48:40.918704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.918503    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g" event={"ID":"1d4e4ee9-0e4d-478f-9a88-8dc4c082150e","Type":"ContainerDied","Data":"ad4451f590527222446c4dc801df4931ac3c9a8ec79405bc4d5dead8439feccd"}
Nov 02 07:48:40.921001 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.920971    4898 generic.go:356] "Generic (PLEG): container finished" podID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerID="f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879" exitCode=0
Nov 02 07:48:40.921152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.921131    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" event={"ID":"06eb45ea-2696-40be-9e1f-90bc0fe6fead","Type":"ContainerDied","Data":"f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879"}
Nov 02 07:48:40.921240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.921178    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk" event={"ID":"06eb45ea-2696-40be-9e1f-90bc0fe6fead","Type":"ContainerDied","Data":"8660bf7001f1b416882e5be88b4c153a8ed75dc23f8d8aabe35bbc60ec4f659e"}
Nov 02 07:48:40.921240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.921229    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"
Nov 02 07:48:40.931505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.931473    4898 scope.go:122] "RemoveContainer" containerID="4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3"
Nov 02 07:48:40.936908 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:48:40.936426    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3\": container with ID starting with 4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3 not found: ID does not exist" containerID="4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3"
Nov 02 07:48:40.936908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.936474    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3"} err="failed to get container status \"4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3\": rpc error: code = NotFound desc = could not find container \"4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3\": container with ID starting with 4e21fc990cce5da56bb715462c099b6ce7d4eeb0c10713a8f855c844768783a3 not found: ID does not exist"
Nov 02 07:48:40.936908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.936520    4898 scope.go:122] "RemoveContainer" containerID="073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071"
Nov 02 07:48:40.951096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.951073    4898 scope.go:122] "RemoveContainer" containerID="073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071"
Nov 02 07:48:40.951449 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:48:40.951426    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071\": container with ID starting with 073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071 not found: ID does not exist" containerID="073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071"
Nov 02 07:48:40.951549 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.951460    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071"} err="failed to get container status \"073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071\": rpc error: code = NotFound desc = could not find container \"073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071\": container with ID starting with 073b1630c4fa9586468c8d81806ba495a9402c9d00c7c5f3da59d2e5c62a5071 not found: ID does not exist"
Nov 02 07:48:40.951549 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.951483    4898 scope.go:122] "RemoveContainer" containerID="f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879"
Nov 02 07:48:40.962358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.962338    4898 scope.go:122] "RemoveContainer" containerID="f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879"
Nov 02 07:48:40.963557 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:48:40.963533    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879\": container with ID starting with f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879 not found: ID does not exist" containerID="f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879"
Nov 02 07:48:40.963654 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:40.963565    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879"} err="failed to get container status \"f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879\": rpc error: code = NotFound desc = could not find container \"f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879\": container with ID starting with f8d3ac2f3f4c2e384d7f6dc0727f2dcc30e3904bb93b17f10c2e0139a8cb7879 not found: ID does not exist"
Nov 02 07:48:41.002254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.002216    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"]
Nov 02 07:48:41.016415 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.016376    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-b652g"]
Nov 02 07:48:41.042872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.042831    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"]
Nov 02 07:48:41.049719 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.049693    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-68twk"]
Nov 02 07:48:41.089941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.089862    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"]
Nov 02 07:48:41.095730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.095700    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k9x79"]
Nov 02 07:48:41.373917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.373819    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"]
Nov 02 07:48:41.374622 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.374598    4898 cpu_manager.go:414] "containerMap: removing container" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy"
Nov 02 07:48:41.374622 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.374621    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy"
Nov 02 07:48:41.374818 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.374642    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" containerName="istio-proxy"
Nov 02 07:48:41.374818 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.374650    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" containerName="istio-proxy"
Nov 02 07:48:41.374818 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.374672    4898 cpu_manager.go:414] "containerMap: removing container" podUID="64107366-d1ff-46e2-8883-f21ae646ef25" containerName="istio-proxy"
Nov 02 07:48:41.374818 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.374679    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64107366-d1ff-46e2-8883-f21ae646ef25" containerName="istio-proxy"
Nov 02 07:48:41.375031 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.374848    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" containerName="istio-proxy"
Nov 02 07:48:41.375031 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.374863    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" containerName="istio-proxy"
Nov 02 07:48:41.375031 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.374874    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="64107366-d1ff-46e2-8883-f21ae646ef25" containerName="istio-proxy"
Nov 02 07:48:41.381041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.380117    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.403846 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.403815    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-eb64ad-openshift-default-dockercfg-422pz\""
Nov 02 07:48:41.404458 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.404350    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"]
Nov 02 07:48:41.466089 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.466053    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06eb45ea-2696-40be-9e1f-90bc0fe6fead" path="/var/lib/kubelet/pods/06eb45ea-2696-40be-9e1f-90bc0fe6fead/volumes"
Nov 02 07:48:41.466653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.466637    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d4e4ee9-0e4d-478f-9a88-8dc4c082150e" path="/var/lib/kubelet/pods/1d4e4ee9-0e4d-478f-9a88-8dc4c082150e/volumes"
Nov 02 07:48:41.467172 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.467148    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64107366-d1ff-46e2-8883-f21ae646ef25" path="/var/lib/kubelet/pods/64107366-d1ff-46e2-8883-f21ae646ef25/volumes"
Nov 02 07:48:41.505276 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.505239    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-certs\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.505428 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.505297    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1d63fb-e287-42c3-8496-237e4556e5f4-istiod-ca-cert\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.505497 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.505416    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh9x2\" (UniqueName: \"kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-kube-api-access-qh9x2\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.505532 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.505508    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-socket\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.505676 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.505657    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-token\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.505805 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.505784    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-envoy\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.505996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.505820    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-credential-socket\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.505996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.505883    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-data\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.505996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.505985    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-podinfo\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.621912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.621103    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-token\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.621912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.621233    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-envoy\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.621912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.621264    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-credential-socket\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.621912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.621287    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-data\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.621912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.621333    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-podinfo\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.621912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.621372    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-certs\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.621912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.621421    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1d63fb-e287-42c3-8496-237e4556e5f4-istiod-ca-cert\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.621912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.621468    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9x2\" (UniqueName: \"kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-kube-api-access-qh9x2\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.621912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.621514    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-socket\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.622538 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.622027    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-socket\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.622538 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.622273    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-certs\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.622659 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.622626    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1d63fb-e287-42c3-8496-237e4556e5f4-istiod-ca-cert\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.626957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.622719    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-data\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.626957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.623464    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-credential-socket\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.641307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.641268    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-envoy\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.653446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.652552    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-podinfo\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.658932 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.656457    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9x2\" (UniqueName: \"kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-kube-api-access-qh9x2\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.684953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.681808    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-token\") pod \"e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") " pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:41.705786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:41.705751    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:42.035004 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:48:42.034942    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1d63fb_e287_42c3_8496_237e4556e5f4.slice/crio-6c54d81f09eac5cd94131bbd39322eb9c766d194119f70a80d1a6c22ad4b6224 WatchSource:0}: Error finding container 6c54d81f09eac5cd94131bbd39322eb9c766d194119f70a80d1a6c22ad4b6224: Status 404 returned error can't find the container with id 6c54d81f09eac5cd94131bbd39322eb9c766d194119f70a80d1a6c22ad4b6224
Nov 02 07:48:42.036749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:42.036723    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"]
Nov 02 07:48:42.039760 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:42.039720    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:42.039875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:42.039803    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:42.039875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:42.039848    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:48:42.954138 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:42.954098    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" event={"ID":"4e1d63fb-e287-42c3-8496-237e4556e5f4","Type":"ContainerStarted","Data":"3d45d340c2f0282df9866496edf469e78c3439d3370e6cf0c98af99e1cdca6c2"}
Nov 02 07:48:42.954661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:42.954194    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" event={"ID":"4e1d63fb-e287-42c3-8496-237e4556e5f4","Type":"ContainerStarted","Data":"6c54d81f09eac5cd94131bbd39322eb9c766d194119f70a80d1a6c22ad4b6224"}
Nov 02 07:48:43.004423 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:43.004359    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" podStartSLOduration=2.004341479 podStartE2EDuration="2.004341479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:42.999637644 +0000 UTC m=+1473.689964609" watchObservedRunningTime="2026-11-02 07:48:43.004341479 +0000 UTC m=+1473.694668444"
Nov 02 07:48:43.707551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:43.707514    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:43.715741 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:43.715672    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:43.963072 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:43.962986    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:43.964137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:43.964108    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:48:44.993000 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:48:44.992957    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:48:47.904376 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:47.904343    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"]
Nov 02 07:48:47.909011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:47.908985    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"
Nov 02 07:48:47.932286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:47.932252    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"]
Nov 02 07:48:47.934695 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:47.934651    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eb64ad\""
Nov 02 07:48:47.962934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:47.962883    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eb64ad-dockercfg-q5qqr\""
Nov 02 07:48:47.969844 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:47.969816    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"]
Nov 02 07:48:47.974155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:47.974132    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"
Nov 02 07:48:48.077936 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.053152    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pzq6\" (UniqueName: \"kubernetes.io/projected/4a18e244-a44b-4da4-b7ff-794ce1c247e2-kube-api-access-9pzq6\") pod \"payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n\" (UID: \"4a18e244-a44b-4da4-b7ff-794ce1c247e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"
Nov 02 07:48:48.077936 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.053368    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a18e244-a44b-4da4-b7ff-794ce1c247e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n\" (UID: \"4a18e244-a44b-4da4-b7ff-794ce1c247e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"
Nov 02 07:48:48.140422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.137930    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"]
Nov 02 07:48:48.155529 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.155285    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6364b413-cb33-4342-91c8-a9d4cea78696-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n\" (UID: \"6364b413-cb33-4342-91c8-a9d4cea78696\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"
Nov 02 07:48:48.155808 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.155781    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pzq6\" (UniqueName: \"kubernetes.io/projected/4a18e244-a44b-4da4-b7ff-794ce1c247e2-kube-api-access-9pzq6\") pod \"payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n\" (UID: \"4a18e244-a44b-4da4-b7ff-794ce1c247e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"
Nov 02 07:48:48.156046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.156028    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nghcb\" (UniqueName: \"kubernetes.io/projected/6364b413-cb33-4342-91c8-a9d4cea78696-kube-api-access-nghcb\") pod \"payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n\" (UID: \"6364b413-cb33-4342-91c8-a9d4cea78696\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"
Nov 02 07:48:48.156346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.156327    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a18e244-a44b-4da4-b7ff-794ce1c247e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n\" (UID: \"4a18e244-a44b-4da4-b7ff-794ce1c247e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"
Nov 02 07:48:48.159043 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.158997    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a18e244-a44b-4da4-b7ff-794ce1c247e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n\" (UID: \"4a18e244-a44b-4da4-b7ff-794ce1c247e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"
Nov 02 07:48:48.183653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.183620    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pzq6\" (UniqueName: \"kubernetes.io/projected/4a18e244-a44b-4da4-b7ff-794ce1c247e2-kube-api-access-9pzq6\") pod \"payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n\" (UID: \"4a18e244-a44b-4da4-b7ff-794ce1c247e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"
Nov 02 07:48:48.229546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.229502    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"
Nov 02 07:48:48.265909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.263610    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6364b413-cb33-4342-91c8-a9d4cea78696-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n\" (UID: \"6364b413-cb33-4342-91c8-a9d4cea78696\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"
Nov 02 07:48:48.265909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.263690    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nghcb\" (UniqueName: \"kubernetes.io/projected/6364b413-cb33-4342-91c8-a9d4cea78696-kube-api-access-nghcb\") pod \"payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n\" (UID: \"6364b413-cb33-4342-91c8-a9d4cea78696\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"
Nov 02 07:48:48.265909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.264689    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6364b413-cb33-4342-91c8-a9d4cea78696-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n\" (UID: \"6364b413-cb33-4342-91c8-a9d4cea78696\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"
Nov 02 07:48:48.297348 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.297319    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nghcb\" (UniqueName: \"kubernetes.io/projected/6364b413-cb33-4342-91c8-a9d4cea78696-kube-api-access-nghcb\") pod \"payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n\" (UID: \"6364b413-cb33-4342-91c8-a9d4cea78696\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"
Nov 02 07:48:48.353515 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.353449    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"
Nov 02 07:48:48.538903 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:48:48.538857    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a18e244_a44b_4da4_b7ff_794ce1c247e2.slice/crio-282411617ec3974f83bc93fed777a1fa71783c320a9579cb207fa62698ed41aa WatchSource:0}: Error finding container 282411617ec3974f83bc93fed777a1fa71783c320a9579cb207fa62698ed41aa: Status 404 returned error can't find the container with id 282411617ec3974f83bc93fed777a1fa71783c320a9579cb207fa62698ed41aa
Nov 02 07:48:48.540828 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.540790    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"]
Nov 02 07:48:48.670009 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:48:48.669944    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6364b413_cb33_4342_91c8_a9d4cea78696.slice/crio-f174f69edc8670f3a0b77e3f2a3fca805d60afccd110d38e7664ab861b1decb6 WatchSource:0}: Error finding container f174f69edc8670f3a0b77e3f2a3fca805d60afccd110d38e7664ab861b1decb6: Status 404 returned error can't find the container with id f174f69edc8670f3a0b77e3f2a3fca805d60afccd110d38e7664ab861b1decb6
Nov 02 07:48:48.672545 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:48.672514    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"]
Nov 02 07:48:49.041124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:49.041030    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n" event={"ID":"6364b413-cb33-4342-91c8-a9d4cea78696","Type":"ContainerStarted","Data":"581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014"}
Nov 02 07:48:49.041124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:49.041091    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n" event={"ID":"6364b413-cb33-4342-91c8-a9d4cea78696","Type":"ContainerStarted","Data":"f174f69edc8670f3a0b77e3f2a3fca805d60afccd110d38e7664ab861b1decb6"}
Nov 02 07:48:49.041124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:49.041119    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"
Nov 02 07:48:49.042813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:49.042786    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n" event={"ID":"4a18e244-a44b-4da4-b7ff-794ce1c247e2","Type":"ContainerStarted","Data":"56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc"}
Nov 02 07:48:49.042988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:49.042843    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"
Nov 02 07:48:49.042988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:49.042867    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n" event={"ID":"4a18e244-a44b-4da4-b7ff-794ce1c247e2","Type":"ContainerStarted","Data":"282411617ec3974f83bc93fed777a1fa71783c320a9579cb207fa62698ed41aa"}
Nov 02 07:48:49.085574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:49.085531    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n" podStartSLOduration=2.085516655 podStartE2EDuration="2.085516655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:49.082606845 +0000 UTC m=+1479.772933828" watchObservedRunningTime="2026-11-02 07:48:49.085516655 +0000 UTC m=+1479.775843622"
Nov 02 07:48:49.127797 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:48:49.127750    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n" podStartSLOduration=2.127736233 podStartE2EDuration="2.127736233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:48:49.124105161 +0000 UTC m=+1479.814432127" watchObservedRunningTime="2026-11-02 07:48:49.127736233 +0000 UTC m=+1479.818063197"
Nov 02 07:49:00.054142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:00.054043    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"
Nov 02 07:49:00.054759 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:00.054731    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"
Nov 02 07:49:08.739370 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.739318    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"]
Nov 02 07:49:08.744901 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.744863    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.759583 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.759554    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-11b2e0-openshift-default-dockercfg-5kml5\""
Nov 02 07:49:08.782416 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.782379    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"]
Nov 02 07:49:08.837508 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.837471    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-token\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.837721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.837528    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-data\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.837835 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.837750    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-credential-socket\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.838014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.837988    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-certs\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.838185 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.838164    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-socket\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.838322 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.838257    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-podinfo\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.838497 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.838475    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-envoy\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.838660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.838529    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a9a6c9-7420-428e-a684-8dba457ecae9-istiod-ca-cert\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.838660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.838556    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tnlt\" (UniqueName: \"kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-kube-api-access-9tnlt\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.939757 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.939720    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-credential-socket\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.939965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.939800    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-certs\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.939965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.939825    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-socket\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.939965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.939854    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-podinfo\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.939965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.939954    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-envoy\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.940193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.939978    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a9a6c9-7420-428e-a684-8dba457ecae9-istiod-ca-cert\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.940251 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.940208    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-credential-socket\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.940309 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.940261    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-certs\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.940309 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.940267    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tnlt\" (UniqueName: \"kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-kube-api-access-9tnlt\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.940469 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.940450    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-token\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.940524 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.940499    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-data\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.940694 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.940672    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a9a6c9-7420-428e-a684-8dba457ecae9-istiod-ca-cert\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.941043 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.941020    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-socket\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.941107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.941061    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-data\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.942346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.942320    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-envoy\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.942851 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.942822    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-podinfo\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.965120 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.965095    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-token\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:08.965350 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:08.965327    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tnlt\" (UniqueName: \"kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-kube-api-access-9tnlt\") pod \"e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") " pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:09.065524 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:09.065172    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:09.453742 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:49:09.453709    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a9a6c9_7420_428e_a684_8dba457ecae9.slice/crio-4260a6c8f949bf2e6f66bcac62477979e729c8f5dac1141dd8d8483349663b9d WatchSource:0}: Error finding container 4260a6c8f949bf2e6f66bcac62477979e729c8f5dac1141dd8d8483349663b9d: Status 404 returned error can't find the container with id 4260a6c8f949bf2e6f66bcac62477979e729c8f5dac1141dd8d8483349663b9d
Nov 02 07:49:09.455065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:09.455039    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"]
Nov 02 07:49:09.456528 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:09.456485    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:09.456633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:09.456573    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:09.456633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:09.456619    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:10.058987 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:10.058960    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:49:10.059489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:10.058961    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:49:10.071653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:10.071626    4898 kubelet.go:1689] "Image garbage collection succeeded"
Nov 02 07:49:10.293208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:10.293156    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" event={"ID":"f8a9a6c9-7420-428e-a684-8dba457ecae9","Type":"ContainerStarted","Data":"ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb"}
Nov 02 07:49:10.293420 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:10.293222    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" event={"ID":"f8a9a6c9-7420-428e-a684-8dba457ecae9","Type":"ContainerStarted","Data":"4260a6c8f949bf2e6f66bcac62477979e729c8f5dac1141dd8d8483349663b9d"}
Nov 02 07:49:10.366030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:10.365930    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" podStartSLOduration=2.365883136 podStartE2EDuration="2.365883136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:49:10.350675006 +0000 UTC m=+1501.041001970" watchObservedRunningTime="2026-11-02 07:49:10.365883136 +0000 UTC m=+1501.056210172"
Nov 02 07:49:11.049373 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:11.048101    4898 scope.go:122] "RemoveContainer" containerID="cb10b9515c6333fe197df997aa39699d8cbb493fa07036ef0791547e8968fb35"
Nov 02 07:49:11.063539 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:11.063512    4898 scope.go:122] "RemoveContainer" containerID="64becaaddcdc711dbad1b09e147c50b17bcca28611a024ad9e625e38c48dfe3b"
Nov 02 07:49:11.066808 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:11.066780    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:11.073341 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:11.073316    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:11.074210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:11.074186    4898 scope.go:122] "RemoveContainer" containerID="288ba54fa86b36b29979d999d38551f242ea4a5a8eb52b044cb1529d2377c925"
Nov 02 07:49:11.303861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:11.303777    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:11.304772 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:11.304747    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:49:15.024399 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:49:15.024358    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:49:15.136201 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:15.136167    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"]
Nov 02 07:49:15.144604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:15.144576    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"
Nov 02 07:49:15.186912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:15.181481    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"]
Nov 02 07:49:15.187823 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:15.187780    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-11b2e0-dockercfg-8d5t2\""
Nov 02 07:49:15.194591 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:15.193733    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-11b2e0\""
Nov 02 07:49:15.203305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:15.195072    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl4nl\" (UniqueName: \"kubernetes.io/projected/060dc90e-d6de-4546-9af1-c0a809e3e8a6-kube-api-access-tl4nl\") pod \"payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk\" (UID: \"060dc90e-d6de-4546-9af1-c0a809e3e8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"
Nov 02 07:49:15.203305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:15.200075    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060dc90e-d6de-4546-9af1-c0a809e3e8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk\" (UID: \"060dc90e-d6de-4546-9af1-c0a809e3e8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"
Nov 02 07:49:15.449314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:15.449273    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl4nl\" (UniqueName: \"kubernetes.io/projected/060dc90e-d6de-4546-9af1-c0a809e3e8a6-kube-api-access-tl4nl\") pod \"payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk\" (UID: \"060dc90e-d6de-4546-9af1-c0a809e3e8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"
Nov 02 07:49:15.449520 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:15.449401    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060dc90e-d6de-4546-9af1-c0a809e3e8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk\" (UID: \"060dc90e-d6de-4546-9af1-c0a809e3e8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"
Nov 02 07:49:15.450496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:15.450469    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060dc90e-d6de-4546-9af1-c0a809e3e8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk\" (UID: \"060dc90e-d6de-4546-9af1-c0a809e3e8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"
Nov 02 07:49:15.508561 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:15.508519    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl4nl\" (UniqueName: \"kubernetes.io/projected/060dc90e-d6de-4546-9af1-c0a809e3e8a6-kube-api-access-tl4nl\") pod \"payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk\" (UID: \"060dc90e-d6de-4546-9af1-c0a809e3e8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"
Nov 02 07:49:15.796349 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:15.796248    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"
Nov 02 07:49:16.109412 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:49:16.109378    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060dc90e_d6de_4546_9af1_c0a809e3e8a6.slice/crio-99ed71905d058d81b522a6ffc63e34fe40b41230430ffd15a063cbf447ac9719 WatchSource:0}: Error finding container 99ed71905d058d81b522a6ffc63e34fe40b41230430ffd15a063cbf447ac9719: Status 404 returned error can't find the container with id 99ed71905d058d81b522a6ffc63e34fe40b41230430ffd15a063cbf447ac9719
Nov 02 07:49:16.110750 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:16.110721    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"]
Nov 02 07:49:16.477811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:16.477769    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk" event={"ID":"060dc90e-d6de-4546-9af1-c0a809e3e8a6","Type":"ContainerStarted","Data":"bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d"}
Nov 02 07:49:16.478122 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:16.478099    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk" event={"ID":"060dc90e-d6de-4546-9af1-c0a809e3e8a6","Type":"ContainerStarted","Data":"99ed71905d058d81b522a6ffc63e34fe40b41230430ffd15a063cbf447ac9719"}
Nov 02 07:49:16.479354 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:16.479331    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"
Nov 02 07:49:16.522210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:16.522143    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk" podStartSLOduration=1.522125354 podStartE2EDuration="1.522125354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:49:16.5204609 +0000 UTC m=+1507.210787866" watchObservedRunningTime="2026-11-02 07:49:16.522125354 +0000 UTC m=+1507.212452318"
Nov 02 07:49:20.853364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:20.853327    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"]
Nov 02 07:49:20.853850 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:20.853556    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp" podUID="3f4791d4-db06-4c5d-8423-745556f551a3" containerName="istio-proxy" containerID="cri-o://ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be" gracePeriod=30
Nov 02 07:49:23.548229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:23.548188    4898 generic.go:356] "Generic (PLEG): container finished" podID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerID="612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea" exitCode=0
Nov 02 07:49:23.548763 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:23.548235    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" event={"ID":"5c749a89-2eb7-4bdf-810a-ffae61b7e264","Type":"ContainerDied","Data":"612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea"}
Nov 02 07:49:23.548999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:23.548974    4898 scope.go:122] "RemoveContainer" containerID="612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea"
Nov 02 07:49:23.551152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:23.551110    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:23.551285 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:23.551224    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:23.553993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:23.551328    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:24.603673 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:24.603630    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" event={"ID":"5c749a89-2eb7-4bdf-810a-ffae61b7e264","Type":"ContainerStarted","Data":"409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a"}
Nov 02 07:49:25.135961 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:25.135927    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:49:25.136992 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:25.136966    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 02 07:49:25.137108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:25.137015    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 02 07:49:26.137092 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.137006    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 02 07:49:26.137092 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.137069    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 02 07:49:26.226250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.226218    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:49:26.253240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.253210    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-istio-token\" (UniqueName: \"kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-istio-token\") pod \"3f4791d4-db06-4c5d-8423-745556f551a3\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") "
Nov 02 07:49:26.253377 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.253278    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-socket\") pod \"3f4791d4-db06-4c5d-8423-745556f551a3\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") "
Nov 02 07:49:26.253377 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.253370    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f4791d4-db06-4c5d-8423-745556f551a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4791d4-db06-4c5d-8423-745556f551a3-istio-podinfo\") pod \"3f4791d4-db06-4c5d-8423-745556f551a3\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") "
Nov 02 07:49:26.254277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.253586    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-envoy\") pod \"3f4791d4-db06-4c5d-8423-745556f551a3\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") "
Nov 02 07:49:26.254277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.253675    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4791d4-db06-4c5d-8423-745556f551a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4791d4-db06-4c5d-8423-745556f551a3-istiod-ca-cert\") pod \"3f4791d4-db06-4c5d-8423-745556f551a3\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") "
Nov 02 07:49:26.254277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.253746    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-data\") pod \"3f4791d4-db06-4c5d-8423-745556f551a3\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") "
Nov 02 07:49:26.254277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.253876    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-certs\") pod \"3f4791d4-db06-4c5d-8423-745556f551a3\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") "
Nov 02 07:49:26.254277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.253910    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-socket" pod "3f4791d4-db06-4c5d-8423-745556f551a3" (UID: "3f4791d4-db06-4c5d-8423-745556f551a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:26.254277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.254020    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-kube-api-access-snthz\" (UniqueName: \"kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-kube-api-access-snthz\") pod \"3f4791d4-db06-4c5d-8423-745556f551a3\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") "
Nov 02 07:49:26.254277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.254060    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-credential-socket\") pod \"3f4791d4-db06-4c5d-8423-745556f551a3\" (UID: \"3f4791d4-db06-4c5d-8423-745556f551a3\") "
Nov 02 07:49:26.254277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.254143    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4791d4-db06-4c5d-8423-745556f551a3-istiod-ca-cert" pod "3f4791d4-db06-4c5d-8423-745556f551a3" (UID: "3f4791d4-db06-4c5d-8423-745556f551a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:26.255000 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.254514    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-credential-socket" pod "3f4791d4-db06-4c5d-8423-745556f551a3" (UID: "3f4791d4-db06-4c5d-8423-745556f551a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:26.255000 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.254531    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-certs" pod "3f4791d4-db06-4c5d-8423-745556f551a3" (UID: "3f4791d4-db06-4c5d-8423-745556f551a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:26.255000 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.254693    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-data" pod "3f4791d4-db06-4c5d-8423-745556f551a3" (UID: "3f4791d4-db06-4c5d-8423-745556f551a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:26.315913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.255368    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:26.315913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.255393    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:26.315913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.255408    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4791d4-db06-4c5d-8423-745556f551a3-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:26.315913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.255421    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:26.315913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.255435    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:26.315913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.297056    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-istio-token" pod "3f4791d4-db06-4c5d-8423-745556f551a3" (UID: "3f4791d4-db06-4c5d-8423-745556f551a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:26.315913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.298057    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-kube-api-access-snthz" pod "3f4791d4-db06-4c5d-8423-745556f551a3" (UID: "3f4791d4-db06-4c5d-8423-745556f551a3"). InnerVolumeSpecName "kube-api-access-snthz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:26.315913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.298135    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-envoy" pod "3f4791d4-db06-4c5d-8423-745556f551a3" (UID: "3f4791d4-db06-4c5d-8423-745556f551a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:49:26.315913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.298188    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f4791d4-db06-4c5d-8423-745556f551a3-istio-podinfo" pod "3f4791d4-db06-4c5d-8423-745556f551a3" (UID: "3f4791d4-db06-4c5d-8423-745556f551a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:49:26.388002 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.387807    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snthz\" (UniqueName: \"kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-kube-api-access-snthz\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:26.388002 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.387843    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4791d4-db06-4c5d-8423-745556f551a3-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:26.388002 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.387860    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4791d4-db06-4c5d-8423-745556f551a3-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:26.388002 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.387873    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4791d4-db06-4c5d-8423-745556f551a3-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:26.694908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.694110    4898 generic.go:356] "Generic (PLEG): container finished" podID="3f4791d4-db06-4c5d-8423-745556f551a3" containerID="ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be" exitCode=0
Nov 02 07:49:26.694908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.694169    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"
Nov 02 07:49:26.694908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.694182    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp" event={"ID":"3f4791d4-db06-4c5d-8423-745556f551a3","Type":"ContainerDied","Data":"ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be"}
Nov 02 07:49:26.694908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.694239    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp" event={"ID":"3f4791d4-db06-4c5d-8423-745556f551a3","Type":"ContainerDied","Data":"e91ecaa402f0e37661683cb4408d94e49c57fa6e0a5021a69c5880f787c82dbb"}
Nov 02 07:49:26.694908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.694263    4898 scope.go:122] "RemoveContainer" containerID="ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be"
Nov 02 07:49:26.707246 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.705411    4898 scope.go:122] "RemoveContainer" containerID="ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be"
Nov 02 07:49:26.707246 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:49:26.705835    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be\": container with ID starting with ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be not found: ID does not exist" containerID="ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be"
Nov 02 07:49:26.707246 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.705868    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be"} err="failed to get container status \"ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be\": rpc error: code = NotFound desc = could not find container \"ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be\": container with ID starting with ce127d50384c08d5835d6fa4d231715d570204565b10f8ae8ae7670de44094be not found: ID does not exist"
Nov 02 07:49:26.751015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.750985    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"]
Nov 02 07:49:26.758911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:26.755153    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-nk2tp"]
Nov 02 07:49:27.136850 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.136767    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 02 07:49:27.137063 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.136848    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 02 07:49:27.150762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.150728    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"]
Nov 02 07:49:27.151438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.151421    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3f4791d4-db06-4c5d-8423-745556f551a3" containerName="istio-proxy"
Nov 02 07:49:27.151493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.151440    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4791d4-db06-4c5d-8423-745556f551a3" containerName="istio-proxy"
Nov 02 07:49:27.151544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.151535    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4791d4-db06-4c5d-8423-745556f551a3" containerName="istio-proxy"
Nov 02 07:49:27.169215 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.169190    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"
Nov 02 07:49:27.189030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.189005    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-677726-dockercfg-2nbv9\""
Nov 02 07:49:27.189265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.189249    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-677726\""
Nov 02 07:49:27.199806 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.199777    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"]
Nov 02 07:49:27.248945 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.248909    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"]
Nov 02 07:49:27.295943 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.268094    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"
Nov 02 07:49:27.312929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.309836    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm\" (UID: \"8a79a41d-1043-4601-ab0b-ee16cbd02f9d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"
Nov 02 07:49:27.312929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.310050    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm4pf\" (UniqueName: \"kubernetes.io/projected/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-kube-api-access-zm4pf\") pod \"payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm\" (UID: \"8a79a41d-1043-4601-ab0b-ee16cbd02f9d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"
Nov 02 07:49:27.414809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.411211    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm\" (UID: \"8a79a41d-1043-4601-ab0b-ee16cbd02f9d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"
Nov 02 07:49:27.414809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.411382    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm4pf\" (UniqueName: \"kubernetes.io/projected/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-kube-api-access-zm4pf\") pod \"payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm\" (UID: \"8a79a41d-1043-4601-ab0b-ee16cbd02f9d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"
Nov 02 07:49:27.414809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.411475    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh\" (UID: \"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"
Nov 02 07:49:27.414809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.411588    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8dqz\" (UniqueName: \"kubernetes.io/projected/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-kube-api-access-b8dqz\") pod \"payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh\" (UID: \"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"
Nov 02 07:49:27.414809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.412738    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm\" (UID: \"8a79a41d-1043-4601-ab0b-ee16cbd02f9d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"
Nov 02 07:49:27.434609 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.430828    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"]
Nov 02 07:49:27.528944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.525521    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh\" (UID: \"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"
Nov 02 07:49:27.528944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.525625    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8dqz\" (UniqueName: \"kubernetes.io/projected/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-kube-api-access-b8dqz\") pod \"payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh\" (UID: \"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"
Nov 02 07:49:27.528944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.526861    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh\" (UID: \"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"
Nov 02 07:49:27.543912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.531749    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm4pf\" (UniqueName: \"kubernetes.io/projected/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-kube-api-access-zm4pf\") pod \"payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm\" (UID: \"8a79a41d-1043-4601-ab0b-ee16cbd02f9d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"
Nov 02 07:49:27.562049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.561996    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8dqz\" (UniqueName: \"kubernetes.io/projected/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-kube-api-access-b8dqz\") pod \"payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh\" (UID: \"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"
Nov 02 07:49:27.608814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.608778    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4791d4-db06-4c5d-8423-745556f551a3" path="/var/lib/kubelet/pods/3f4791d4-db06-4c5d-8423-745556f551a3/volumes"
Nov 02 07:49:27.609477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.609451    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"
Nov 02 07:49:27.802910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.802260    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"
Nov 02 07:49:27.995235 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:27.994022    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"]
Nov 02 07:49:27.997546 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:49:27.997514    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7b4fa4_1a09_46cb_97e3_f9feed6ab765.slice/crio-ccf360aecfea29de871ffc0b6c9b98a09548faa00673197f05572c85705cc1d6 WatchSource:0}: Error finding container ccf360aecfea29de871ffc0b6c9b98a09548faa00673197f05572c85705cc1d6: Status 404 returned error can't find the container with id ccf360aecfea29de871ffc0b6c9b98a09548faa00673197f05572c85705cc1d6
Nov 02 07:49:28.136416 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:28.136376    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 02 07:49:28.136530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:28.136434    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 02 07:49:28.193093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:28.193058    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"]
Nov 02 07:49:28.193849 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:49:28.193816    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a79a41d_1043_4601_ab0b_ee16cbd02f9d.slice/crio-6b44ac0add256cf9c23f47d1124c9cdad53f1e22afff014276401feb6fee0dc8 WatchSource:0}: Error finding container 6b44ac0add256cf9c23f47d1124c9cdad53f1e22afff014276401feb6fee0dc8: Status 404 returned error can't find the container with id 6b44ac0add256cf9c23f47d1124c9cdad53f1e22afff014276401feb6fee0dc8
Nov 02 07:49:28.509758 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:28.509672    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"
Nov 02 07:49:28.741042 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:28.741000    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm" event={"ID":"8a79a41d-1043-4601-ab0b-ee16cbd02f9d","Type":"ContainerStarted","Data":"b450509ed37d972df3319a1124a81276989ea353dd70644195e9765cd2058a75"}
Nov 02 07:49:28.741207 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:28.741073    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm" event={"ID":"8a79a41d-1043-4601-ab0b-ee16cbd02f9d","Type":"ContainerStarted","Data":"6b44ac0add256cf9c23f47d1124c9cdad53f1e22afff014276401feb6fee0dc8"}
Nov 02 07:49:28.742182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:28.742160    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"
Nov 02 07:49:28.744406 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:28.744376    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh" event={"ID":"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765","Type":"ContainerStarted","Data":"d6c8568b2c596c1a1bf0165b964dafc69e9e4593014c23aa4227e2e282b9de19"}
Nov 02 07:49:28.744526 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:28.744427    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh" event={"ID":"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765","Type":"ContainerStarted","Data":"ccf360aecfea29de871ffc0b6c9b98a09548faa00673197f05572c85705cc1d6"}
Nov 02 07:49:28.745314 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:28.745296    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"
Nov 02 07:49:28.818964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:28.818242    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm" podStartSLOduration=1.818220521 podStartE2EDuration="1.818220521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:49:28.815308507 +0000 UTC m=+1519.505635496" watchObservedRunningTime="2026-11-02 07:49:28.818220521 +0000 UTC m=+1519.508547486"
Nov 02 07:49:28.833018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:28.832983    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:49:29.136234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:29.136158    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 02 07:49:29.136234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:29.136213    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 02 07:49:30.184914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:30.178867    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:49:30.199952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:30.189152    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:49:30.250924 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:30.249332    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh" podStartSLOduration=3.249310517 podStartE2EDuration="3.249310517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:49:28.869138318 +0000 UTC m=+1519.559465307" watchObservedRunningTime="2026-11-02 07:49:30.249310517 +0000 UTC m=+1520.939637482"
Nov 02 07:49:38.645257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.645218    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"]
Nov 02 07:49:38.649838 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.649809    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:38.681009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.676279    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2296317e-nmtzc\" (UID: \"6917528e-6e93-4266-b4e1-1e3a99f75195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:38.681009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.676346    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flh84\" (UniqueName: \"kubernetes.io/projected/6917528e-6e93-4266-b4e1-1e3a99f75195-kube-api-access-flh84\") pod \"maas-api-revoke-sub-2296317e-nmtzc\" (UID: \"6917528e-6e93-4266-b4e1-1e3a99f75195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:38.681009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.676445    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2296317e-nmtzc\" (UID: \"6917528e-6e93-4266-b4e1-1e3a99f75195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:38.681009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.676847    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:49:38.681009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.676959    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:49:38.681009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.676990    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:49:38.681009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.677030    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:49:38.681009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.677870    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"]
Nov 02 07:49:38.777799 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.777753    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2296317e-nmtzc\" (UID: \"6917528e-6e93-4266-b4e1-1e3a99f75195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:38.778135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.778111    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flh84\" (UniqueName: \"kubernetes.io/projected/6917528e-6e93-4266-b4e1-1e3a99f75195-kube-api-access-flh84\") pod \"maas-api-revoke-sub-2296317e-nmtzc\" (UID: \"6917528e-6e93-4266-b4e1-1e3a99f75195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:38.778321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.778306    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2296317e-nmtzc\" (UID: \"6917528e-6e93-4266-b4e1-1e3a99f75195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:38.778743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.778711    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2296317e-nmtzc\" (UID: \"6917528e-6e93-4266-b4e1-1e3a99f75195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:38.781743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.781694    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2296317e-nmtzc\" (UID: \"6917528e-6e93-4266-b4e1-1e3a99f75195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:38.831912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.829656    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flh84\" (UniqueName: \"kubernetes.io/projected/6917528e-6e93-4266-b4e1-1e3a99f75195-kube-api-access-flh84\") pod \"maas-api-revoke-sub-2296317e-nmtzc\" (UID: \"6917528e-6e93-4266-b4e1-1e3a99f75195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:38.971103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:38.971066    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:39.287456 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:39.287426    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"]
Nov 02 07:49:39.767262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:39.767216    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"
Nov 02 07:49:39.932531 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:39.932497    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc" event={"ID":"6917528e-6e93-4266-b4e1-1e3a99f75195","Type":"ContainerStarted","Data":"8f79e031a73df28c9feb5a30c1df185f512bd6162abe3c31f5383c7007d528ad"}
Nov 02 07:49:39.932714 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:39.932553    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc" event={"ID":"6917528e-6e93-4266-b4e1-1e3a99f75195","Type":"ContainerStarted","Data":"62da40155019eeea48841f036009cdc6cedcc918a6cc7666be3cc6801f7f58d4"}
Nov 02 07:49:40.764009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:40.763934    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"
Nov 02 07:49:40.806653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:40.806594    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc" podStartSLOduration=2.806577077 podStartE2EDuration="2.806577077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:49:40.001979504 +0000 UTC m=+1530.692306484" watchObservedRunningTime="2026-11-02 07:49:40.806577077 +0000 UTC m=+1531.496904045"
Nov 02 07:49:40.942791 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:40.942758    4898 generic.go:356] "Generic (PLEG): container finished" podID="6917528e-6e93-4266-b4e1-1e3a99f75195" containerID="8f79e031a73df28c9feb5a30c1df185f512bd6162abe3c31f5383c7007d528ad" exitCode=0
Nov 02 07:49:40.942962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:40.942819    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc" event={"ID":"6917528e-6e93-4266-b4e1-1e3a99f75195","Type":"ContainerDied","Data":"8f79e031a73df28c9feb5a30c1df185f512bd6162abe3c31f5383c7007d528ad"}
Nov 02 07:49:41.418930 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:41.416737    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"]
Nov 02 07:49:41.452865 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:41.452391    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"]
Nov 02 07:49:41.954952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:41.954869    4898 generic.go:356] "Generic (PLEG): container finished" podID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerID="3d45d340c2f0282df9866496edf469e78c3439d3370e6cf0c98af99e1cdca6c2" exitCode=0
Nov 02 07:49:41.954952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:41.954946    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" event={"ID":"4e1d63fb-e287-42c3-8496-237e4556e5f4","Type":"ContainerDied","Data":"3d45d340c2f0282df9866496edf469e78c3439d3370e6cf0c98af99e1cdca6c2"}
Nov 02 07:49:41.955455 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:41.955402    4898 scope.go:122] "RemoveContainer" containerID="3d45d340c2f0282df9866496edf469e78c3439d3370e6cf0c98af99e1cdca6c2"
Nov 02 07:49:41.957948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:41.957919    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:41.958045 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:41.957988    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:41.958045 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:41.958021    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:49:42.318152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.317882    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:42.433874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.433842    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6917528e-6e93-4266-b4e1-1e3a99f75195-kube-api-access-flh84\" (UniqueName: \"kubernetes.io/projected/6917528e-6e93-4266-b4e1-1e3a99f75195-kube-api-access-flh84\") pod \"6917528e-6e93-4266-b4e1-1e3a99f75195\" (UID: \"6917528e-6e93-4266-b4e1-1e3a99f75195\") "
Nov 02 07:49:42.434176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.433945    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-openshift-ca\") pod \"6917528e-6e93-4266-b4e1-1e3a99f75195\" (UID: \"6917528e-6e93-4266-b4e1-1e3a99f75195\") "
Nov 02 07:49:42.434176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.434076    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-xks-ca\") pod \"6917528e-6e93-4266-b4e1-1e3a99f75195\" (UID: \"6917528e-6e93-4266-b4e1-1e3a99f75195\") "
Nov 02 07:49:42.434388 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.434359    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-openshift-ca" pod "6917528e-6e93-4266-b4e1-1e3a99f75195" (UID: "6917528e-6e93-4266-b4e1-1e3a99f75195"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:42.434811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.434785    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:42.436619 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.436593    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-xks-ca" pod "6917528e-6e93-4266-b4e1-1e3a99f75195" (UID: "6917528e-6e93-4266-b4e1-1e3a99f75195"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:49:42.436829 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.436802    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6917528e-6e93-4266-b4e1-1e3a99f75195-kube-api-access-flh84" pod "6917528e-6e93-4266-b4e1-1e3a99f75195" (UID: "6917528e-6e93-4266-b4e1-1e3a99f75195"). InnerVolumeSpecName "kube-api-access-flh84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:42.537220 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.537105    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flh84\" (UniqueName: \"kubernetes.io/projected/6917528e-6e93-4266-b4e1-1e3a99f75195-kube-api-access-flh84\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:42.537472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.537449    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6917528e-6e93-4266-b4e1-1e3a99f75195-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:42.994381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.994339    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" event={"ID":"4e1d63fb-e287-42c3-8496-237e4556e5f4","Type":"ContainerStarted","Data":"4d083dbf6877c556b912ca32ecff68020f4ed489889ed3dadc5a6e8caf95a5f3"}
Nov 02 07:49:42.995820 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.995787    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc" event={"ID":"6917528e-6e93-4266-b4e1-1e3a99f75195","Type":"ContainerDied","Data":"62da40155019eeea48841f036009cdc6cedcc918a6cc7666be3cc6801f7f58d4"}
Nov 02 07:49:42.995954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.995795    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"
Nov 02 07:49:42.995954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:42.995884    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62da40155019eeea48841f036009cdc6cedcc918a6cc7666be3cc6801f7f58d4"
Nov 02 07:49:43.707158 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:43.707116    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:49:43.708265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:43.708239    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 07:49:43.708389 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:43.708290    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 07:49:43.966053 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:43.965964    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:49:44.706800 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:44.706763    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 07:49:44.707236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:44.706823    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 07:49:45.014465 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:49:45.014371    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:49:45.707060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:45.707027    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 07:49:45.707496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:45.707081    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 07:49:46.417106 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:46.417063    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n" podUID="6364b413-cb33-4342-91c8-a9d4cea78696" containerName="payload-processing" containerID="cri-o://581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014" gracePeriod=25
Nov 02 07:49:46.452812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:46.452782    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n" podUID="4a18e244-a44b-4da4-b7ff-794ce1c247e2" containerName="payload-pre-processing" containerID="cri-o://56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc" gracePeriod=25
Nov 02 07:49:46.724388 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:46.714623    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 07:49:46.724388 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:46.714685    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 07:49:46.913848 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:46.913811    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"
Nov 02 07:49:47.030144 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.030113    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"
Nov 02 07:49:47.032623 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.032593    4898 generic.go:356] "Generic (PLEG): container finished" podID="6364b413-cb33-4342-91c8-a9d4cea78696" containerID="581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014" exitCode=0
Nov 02 07:49:47.032835 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.032700    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n" event={"ID":"6364b413-cb33-4342-91c8-a9d4cea78696","Type":"ContainerDied","Data":"581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014"}
Nov 02 07:49:47.032996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.032911    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n" event={"ID":"6364b413-cb33-4342-91c8-a9d4cea78696","Type":"ContainerDied","Data":"f174f69edc8670f3a0b77e3f2a3fca805d60afccd110d38e7664ab861b1decb6"}
Nov 02 07:49:47.032996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.032731    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"
Nov 02 07:49:47.032996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.032948    4898 scope.go:122] "RemoveContainer" containerID="581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014"
Nov 02 07:49:47.034745 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.034725    4898 generic.go:356] "Generic (PLEG): container finished" podID="4a18e244-a44b-4da4-b7ff-794ce1c247e2" containerID="56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc" exitCode=0
Nov 02 07:49:47.035089 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.034949    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"
Nov 02 07:49:47.035306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.034966    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n" event={"ID":"4a18e244-a44b-4da4-b7ff-794ce1c247e2","Type":"ContainerDied","Data":"56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc"}
Nov 02 07:49:47.035435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.035421    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n" event={"ID":"4a18e244-a44b-4da4-b7ff-794ce1c247e2","Type":"ContainerDied","Data":"282411617ec3974f83bc93fed777a1fa71783c320a9579cb207fa62698ed41aa"}
Nov 02 07:49:47.085313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.084116    4898 scope.go:122] "RemoveContainer" containerID="581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014"
Nov 02 07:49:47.120216 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:49:47.118747    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014\": container with ID starting with 581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014 not found: ID does not exist" containerID="581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014"
Nov 02 07:49:47.120216 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.118812    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014"} err="failed to get container status \"581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014\": rpc error: code = NotFound desc = could not find container \"581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014\": container with ID starting with 581ddd890215506c687d4672c26d6a7273730ba450024c5d477f7d7b508db014 not found: ID does not exist"
Nov 02 07:49:47.120216 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.118841    4898 scope.go:122] "RemoveContainer" containerID="56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc"
Nov 02 07:49:47.134983 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.122225    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6364b413-cb33-4342-91c8-a9d4cea78696-kube-api-access-nghcb\" (UniqueName: \"kubernetes.io/projected/6364b413-cb33-4342-91c8-a9d4cea78696-kube-api-access-nghcb\") pod \"6364b413-cb33-4342-91c8-a9d4cea78696\" (UID: \"6364b413-cb33-4342-91c8-a9d4cea78696\") "
Nov 02 07:49:47.134983 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.122339    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6364b413-cb33-4342-91c8-a9d4cea78696-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6364b413-cb33-4342-91c8-a9d4cea78696-plugins-config-volume\") pod \"6364b413-cb33-4342-91c8-a9d4cea78696\" (UID: \"6364b413-cb33-4342-91c8-a9d4cea78696\") "
Nov 02 07:49:47.134983 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.123541    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6364b413-cb33-4342-91c8-a9d4cea78696-plugins-config-volume" pod "6364b413-cb33-4342-91c8-a9d4cea78696" (UID: "6364b413-cb33-4342-91c8-a9d4cea78696"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:47.134983 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.126592    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6364b413-cb33-4342-91c8-a9d4cea78696-kube-api-access-nghcb" pod "6364b413-cb33-4342-91c8-a9d4cea78696" (UID: "6364b413-cb33-4342-91c8-a9d4cea78696"). InnerVolumeSpecName "kube-api-access-nghcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:47.146510 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.146490    4898 scope.go:122] "RemoveContainer" containerID="56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc"
Nov 02 07:49:47.146993 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:49:47.146968    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc\": container with ID starting with 56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc not found: ID does not exist" containerID="56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc"
Nov 02 07:49:47.147100 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.147002    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc"} err="failed to get container status \"56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc\": rpc error: code = NotFound desc = could not find container \"56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc\": container with ID starting with 56894a800110a96e2b765a773430989ef36df2d71e4d12707e89f96a959438bc not found: ID does not exist"
Nov 02 07:49:47.227977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.225860    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a18e244-a44b-4da4-b7ff-794ce1c247e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a18e244-a44b-4da4-b7ff-794ce1c247e2-plugins-config-volume\") pod \"4a18e244-a44b-4da4-b7ff-794ce1c247e2\" (UID: \"4a18e244-a44b-4da4-b7ff-794ce1c247e2\") "
Nov 02 07:49:47.244968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.228331    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a18e244-a44b-4da4-b7ff-794ce1c247e2-kube-api-access-9pzq6\" (UniqueName: \"kubernetes.io/projected/4a18e244-a44b-4da4-b7ff-794ce1c247e2-kube-api-access-9pzq6\") pod \"4a18e244-a44b-4da4-b7ff-794ce1c247e2\" (UID: \"4a18e244-a44b-4da4-b7ff-794ce1c247e2\") "
Nov 02 07:49:47.258914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.245881    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6364b413-cb33-4342-91c8-a9d4cea78696-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:47.278916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.267673    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nghcb\" (UniqueName: \"kubernetes.io/projected/6364b413-cb33-4342-91c8-a9d4cea78696-kube-api-access-nghcb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:47.278916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.267342    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a18e244-a44b-4da4-b7ff-794ce1c247e2-plugins-config-volume" pod "4a18e244-a44b-4da4-b7ff-794ce1c247e2" (UID: "4a18e244-a44b-4da4-b7ff-794ce1c247e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:47.302912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.290602    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a18e244-a44b-4da4-b7ff-794ce1c247e2-kube-api-access-9pzq6" pod "4a18e244-a44b-4da4-b7ff-794ce1c247e2" (UID: "4a18e244-a44b-4da4-b7ff-794ce1c247e2"). InnerVolumeSpecName "kube-api-access-9pzq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:47.371009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.370969    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a18e244-a44b-4da4-b7ff-794ce1c247e2-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:47.371009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.371008    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pzq6\" (UniqueName: \"kubernetes.io/projected/4a18e244-a44b-4da4-b7ff-794ce1c247e2-kube-api-access-9pzq6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:47.415926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.415866    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"]
Nov 02 07:49:47.419562 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.419534    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb64ad-69486cb6f5-znh9n"]
Nov 02 07:49:47.452947 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.452916    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"]
Nov 02 07:49:47.474911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.472449    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a18e244-a44b-4da4-b7ff-794ce1c247e2" path="/var/lib/kubelet/pods/4a18e244-a44b-4da4-b7ff-794ce1c247e2/volumes"
Nov 02 07:49:47.482948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.478711    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb64ad-69b9695ff-kkw2n"]
Nov 02 07:49:47.706834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.706794    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 07:49:47.707028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:47.706850    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 07:49:48.711006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:48.710973    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:49:48.712080 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:48.712059    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:49:49.464852 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:49.464808    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6364b413-cb33-4342-91c8-a9d4cea78696" path="/var/lib/kubelet/pods/6364b413-cb33-4342-91c8-a9d4cea78696/volumes"
Nov 02 07:49:55.469755 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.469715    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"]
Nov 02 07:49:55.470653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.470631    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6364b413-cb33-4342-91c8-a9d4cea78696" containerName="payload-processing"
Nov 02 07:49:55.470705 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.470656    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6364b413-cb33-4342-91c8-a9d4cea78696" containerName="payload-processing"
Nov 02 07:49:55.470705 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.470696    4898 cpu_manager.go:414] "containerMap: removing container" podUID="4a18e244-a44b-4da4-b7ff-794ce1c247e2" containerName="payload-pre-processing"
Nov 02 07:49:55.470774 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.470705    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a18e244-a44b-4da4-b7ff-794ce1c247e2" containerName="payload-pre-processing"
Nov 02 07:49:55.470774 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.470725    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6917528e-6e93-4266-b4e1-1e3a99f75195" containerName="revoke-keys"
Nov 02 07:49:55.470774 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.470733    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6917528e-6e93-4266-b4e1-1e3a99f75195" containerName="revoke-keys"
Nov 02 07:49:55.470934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.470921    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6364b413-cb33-4342-91c8-a9d4cea78696" containerName="payload-processing"
Nov 02 07:49:55.470976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.470940    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a18e244-a44b-4da4-b7ff-794ce1c247e2" containerName="payload-pre-processing"
Nov 02 07:49:55.470976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.470957    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6917528e-6e93-4266-b4e1-1e3a99f75195" containerName="revoke-keys"
Nov 02 07:49:55.476427 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.476406    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:49:55.499271 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.499238    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"]
Nov 02 07:49:55.502589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.502551    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:49:55.502710 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.502620    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:49:55.502934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.502917    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:49:55.503848 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.503831    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:49:55.630172 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.630131    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dzf5\" (UniqueName: \"kubernetes.io/projected/644352e9-29ba-478b-b055-f8405b740fd6-kube-api-access-2dzf5\") pod \"maas-api-revoke-sub-d54e3970-fsvnd\" (UID: \"644352e9-29ba-478b-b055-f8405b740fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:49:55.630341 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.630298    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d54e3970-fsvnd\" (UID: \"644352e9-29ba-478b-b055-f8405b740fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:49:55.630433 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.630401    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d54e3970-fsvnd\" (UID: \"644352e9-29ba-478b-b055-f8405b740fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:49:55.731696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.731607    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dzf5\" (UniqueName: \"kubernetes.io/projected/644352e9-29ba-478b-b055-f8405b740fd6-kube-api-access-2dzf5\") pod \"maas-api-revoke-sub-d54e3970-fsvnd\" (UID: \"644352e9-29ba-478b-b055-f8405b740fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:49:55.731854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.731706    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d54e3970-fsvnd\" (UID: \"644352e9-29ba-478b-b055-f8405b740fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:49:55.731854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.731769    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d54e3970-fsvnd\" (UID: \"644352e9-29ba-478b-b055-f8405b740fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:49:55.734077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.732958    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d54e3970-fsvnd\" (UID: \"644352e9-29ba-478b-b055-f8405b740fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:49:55.743356 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.743329    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d54e3970-fsvnd\" (UID: \"644352e9-29ba-478b-b055-f8405b740fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:49:55.756062 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.756037    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dzf5\" (UniqueName: \"kubernetes.io/projected/644352e9-29ba-478b-b055-f8405b740fd6-kube-api-access-2dzf5\") pod \"maas-api-revoke-sub-d54e3970-fsvnd\" (UID: \"644352e9-29ba-478b-b055-f8405b740fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:49:55.794286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:55.794246    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:49:56.102323 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:56.102291    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"]
Nov 02 07:49:56.148452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:56.148410    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd" event={"ID":"644352e9-29ba-478b-b055-f8405b740fd6","Type":"ContainerStarted","Data":"aabb432fd59ad59c2190365b376ca0acc718dabf01e708ad9ede11fc85806e6b"}
Nov 02 07:49:57.157933 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:57.157859    4898 generic.go:356] "Generic (PLEG): container finished" podID="644352e9-29ba-478b-b055-f8405b740fd6" containerID="18ebb6cf3b5a36a34a423b0ea332f4a87d91b1876906716818aec2c37a0546ce" exitCode=0
Nov 02 07:49:57.158361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:57.157924    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd" event={"ID":"644352e9-29ba-478b-b055-f8405b740fd6","Type":"ContainerDied","Data":"18ebb6cf3b5a36a34a423b0ea332f4a87d91b1876906716818aec2c37a0546ce"}
Nov 02 07:49:58.557502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:58.557471    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:49:58.588683 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:58.588655    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-xks-ca\") pod \"644352e9-29ba-478b-b055-f8405b740fd6\" (UID: \"644352e9-29ba-478b-b055-f8405b740fd6\") "
Nov 02 07:49:58.588815 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:58.588754    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-openshift-ca\") pod \"644352e9-29ba-478b-b055-f8405b740fd6\" (UID: \"644352e9-29ba-478b-b055-f8405b740fd6\") "
Nov 02 07:49:58.588971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:58.588944    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644352e9-29ba-478b-b055-f8405b740fd6-kube-api-access-2dzf5\" (UniqueName: \"kubernetes.io/projected/644352e9-29ba-478b-b055-f8405b740fd6-kube-api-access-2dzf5\") pod \"644352e9-29ba-478b-b055-f8405b740fd6\" (UID: \"644352e9-29ba-478b-b055-f8405b740fd6\") "
Nov 02 07:49:58.589170 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:58.589135    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-openshift-ca" pod "644352e9-29ba-478b-b055-f8405b740fd6" (UID: "644352e9-29ba-478b-b055-f8405b740fd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:49:58.589811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:58.589573    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:58.591180 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:58.591150    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-xks-ca" pod "644352e9-29ba-478b-b055-f8405b740fd6" (UID: "644352e9-29ba-478b-b055-f8405b740fd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:49:58.591180 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:58.591167    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644352e9-29ba-478b-b055-f8405b740fd6-kube-api-access-2dzf5" pod "644352e9-29ba-478b-b055-f8405b740fd6" (UID: "644352e9-29ba-478b-b055-f8405b740fd6"). InnerVolumeSpecName "kube-api-access-2dzf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:49:58.691539 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:58.691503    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644352e9-29ba-478b-b055-f8405b740fd6-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:58.691539 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:58.691534    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dzf5\" (UniqueName: \"kubernetes.io/projected/644352e9-29ba-478b-b055-f8405b740fd6-kube-api-access-2dzf5\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:49:59.223259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:59.223213    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd" event={"ID":"644352e9-29ba-478b-b055-f8405b740fd6","Type":"ContainerDied","Data":"aabb432fd59ad59c2190365b376ca0acc718dabf01e708ad9ede11fc85806e6b"}
Nov 02 07:49:59.223526 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:59.223330    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aabb432fd59ad59c2190365b376ca0acc718dabf01e708ad9ede11fc85806e6b"
Nov 02 07:49:59.223526 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:49:59.223405    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"
Nov 02 07:50:08.452647 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.452610    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"]
Nov 02 07:50:08.455064 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.453258    4898 cpu_manager.go:414] "containerMap: removing container" podUID="644352e9-29ba-478b-b055-f8405b740fd6" containerName="revoke-keys"
Nov 02 07:50:08.455064 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.453271    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644352e9-29ba-478b-b055-f8405b740fd6" containerName="revoke-keys"
Nov 02 07:50:08.455064 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.453403    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="644352e9-29ba-478b-b055-f8405b740fd6" containerName="revoke-keys"
Nov 02 07:50:08.457073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.457049    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:08.477216 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.477190    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"]
Nov 02 07:50:08.483272 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.483245    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:50:08.490206 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.490117    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:50:08.490206 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.490153    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:50:08.490466 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.490181    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:50:08.574612 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.574573    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab8356e1-64f5b\" (UID: \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:08.574790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.574628    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab8356e1-64f5b\" (UID: \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:08.575118 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.575085    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh798\" (UniqueName: \"kubernetes.io/projected/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-kube-api-access-mh798\") pod \"maas-api-revoke-sub-ab8356e1-64f5b\" (UID: \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:08.680749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.680708    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh798\" (UniqueName: \"kubernetes.io/projected/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-kube-api-access-mh798\") pod \"maas-api-revoke-sub-ab8356e1-64f5b\" (UID: \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:08.681059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.680797    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab8356e1-64f5b\" (UID: \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:08.681059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.680836    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab8356e1-64f5b\" (UID: \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:08.681667 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.681638    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab8356e1-64f5b\" (UID: \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:08.684034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.683996    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab8356e1-64f5b\" (UID: \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:08.705696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.705618    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh798\" (UniqueName: \"kubernetes.io/projected/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-kube-api-access-mh798\") pod \"maas-api-revoke-sub-ab8356e1-64f5b\" (UID: \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:08.779326 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:08.779287    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:09.084495 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:50:09.084452    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2c837d_6c5d_4f3c_ad59_7503f94c997e.slice/crio-d2b1f12eb3cb12e527c23b99626927424749fa05488c4024a33824e98b08d766 WatchSource:0}: Error finding container d2b1f12eb3cb12e527c23b99626927424749fa05488c4024a33824e98b08d766: Status 404 returned error can't find the container with id d2b1f12eb3cb12e527c23b99626927424749fa05488c4024a33824e98b08d766
Nov 02 07:50:09.085313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:09.085290    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"]
Nov 02 07:50:09.350703 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:09.350606    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b" event={"ID":"bc2c837d-6c5d-4f3c-ad59-7503f94c997e","Type":"ContainerStarted","Data":"d2b1f12eb3cb12e527c23b99626927424749fa05488c4024a33824e98b08d766"}
Nov 02 07:50:09.352303 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:09.352269    4898 generic.go:356] "Generic (PLEG): container finished" podID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerID="ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb" exitCode=0
Nov 02 07:50:09.352422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:09.352326    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" event={"ID":"f8a9a6c9-7420-428e-a684-8dba457ecae9","Type":"ContainerDied","Data":"ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb"}
Nov 02 07:50:09.352761 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:09.352747    4898 scope.go:122] "RemoveContainer" containerID="ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb"
Nov 02 07:50:09.354438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:09.354408    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:09.354535 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:09.354482    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:09.354535 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:09.354514    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:10.388534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:10.388453    4898 generic.go:356] "Generic (PLEG): container finished" podID="bc2c837d-6c5d-4f3c-ad59-7503f94c997e" containerID="ca2365921dd3e74c84a27425d075c201fa6dc25c99a5e0f19e3a8c39db2a328e" exitCode=0
Nov 02 07:50:10.389078 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:10.388544    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b" event={"ID":"bc2c837d-6c5d-4f3c-ad59-7503f94c997e","Type":"ContainerDied","Data":"ca2365921dd3e74c84a27425d075c201fa6dc25c99a5e0f19e3a8c39db2a328e"}
Nov 02 07:50:10.390951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:10.390919    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" event={"ID":"f8a9a6c9-7420-428e-a684-8dba457ecae9","Type":"ContainerStarted","Data":"7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc"}
Nov 02 07:50:11.077916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.077265    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:50:11.080623 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.080162    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 07:50:11.080623 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.080233    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 07:50:11.151994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.151958    4898 scope.go:122] "RemoveContainer" containerID="f578a22fdcbc0d1a63976acd42cfa9a4a12b211239c867f78658049016136bf7"
Nov 02 07:50:11.304233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.304203    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:50:11.640690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.640661    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:11.732305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.732262    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-openshift-ca\") pod \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\" (UID: \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\") "
Nov 02 07:50:11.732523 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.732427    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-xks-ca\") pod \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\" (UID: \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\") "
Nov 02 07:50:11.732523 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.732521    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-kube-api-access-mh798\" (UniqueName: \"kubernetes.io/projected/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-kube-api-access-mh798\") pod \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\" (UID: \"bc2c837d-6c5d-4f3c-ad59-7503f94c997e\") "
Nov 02 07:50:11.733482 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.733452    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-openshift-ca" pod "bc2c837d-6c5d-4f3c-ad59-7503f94c997e" (UID: "bc2c837d-6c5d-4f3c-ad59-7503f94c997e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:11.735188 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.735163    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-kube-api-access-mh798" pod "bc2c837d-6c5d-4f3c-ad59-7503f94c997e" (UID: "bc2c837d-6c5d-4f3c-ad59-7503f94c997e"). InnerVolumeSpecName "kube-api-access-mh798". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:11.736012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.735986    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-xks-ca" pod "bc2c837d-6c5d-4f3c-ad59-7503f94c997e" (UID: "bc2c837d-6c5d-4f3c-ad59-7503f94c997e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:11.834649 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.834616    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh798\" (UniqueName: \"kubernetes.io/projected/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-kube-api-access-mh798\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:11.834649 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.834646    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:11.834649 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:11.834656    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc2c837d-6c5d-4f3c-ad59-7503f94c997e-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:12.075804 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:12.075299    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 07:50:12.075804 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:12.075376    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 07:50:12.407801 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:12.407771    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"
Nov 02 07:50:12.407984 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:12.407806    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b" event={"ID":"bc2c837d-6c5d-4f3c-ad59-7503f94c997e","Type":"ContainerDied","Data":"d2b1f12eb3cb12e527c23b99626927424749fa05488c4024a33824e98b08d766"}
Nov 02 07:50:12.407984 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:12.407855    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2b1f12eb3cb12e527c23b99626927424749fa05488c4024a33824e98b08d766"
Nov 02 07:50:13.078107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:13.068989    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 07:50:13.078107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:13.069077    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 07:50:14.068414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:14.068371    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 07:50:14.068706 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:14.068678    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 07:50:15.008343 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:50:15.008302    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:50:15.072925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:15.072720    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 07:50:15.072925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:15.072785    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 07:50:16.071043 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:16.071008    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:50:16.072392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:16.072357    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:50:16.647799 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:16.647760    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"]
Nov 02 07:50:16.648077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:16.648046    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerName="istio-proxy" containerID="cri-o://409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a" gracePeriod=30
Nov 02 07:50:20.918520 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:20.918489    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"]
Nov 02 07:50:20.919141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:20.919126    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bc2c837d-6c5d-4f3c-ad59-7503f94c997e" containerName="revoke-keys"
Nov 02 07:50:20.919195 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:20.919142    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2c837d-6c5d-4f3c-ad59-7503f94c997e" containerName="revoke-keys"
Nov 02 07:50:20.919291 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:20.919282    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2c837d-6c5d-4f3c-ad59-7503f94c997e" containerName="revoke-keys"
Nov 02 07:50:20.923129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:20.923109    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:20.938084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:20.938062    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:50:20.948930 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:20.942024    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"]
Nov 02 07:50:20.948930 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:20.948450    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:50:20.948930 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:20.948540    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:50:20.948930 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:20.948584    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:50:21.011914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.011841    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hl6\" (UniqueName: \"kubernetes.io/projected/d71db5b5-4f54-43ab-ba8d-05e114104d7d-kube-api-access-85hl6\") pod \"maas-api-revoke-sub-4b9c3b2d-c8s9p\" (UID: \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:21.012069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.012038    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b9c3b2d-c8s9p\" (UID: \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:21.012186 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.012165    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b9c3b2d-c8s9p\" (UID: \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:21.114040 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.114008    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85hl6\" (UniqueName: \"kubernetes.io/projected/d71db5b5-4f54-43ab-ba8d-05e114104d7d-kube-api-access-85hl6\") pod \"maas-api-revoke-sub-4b9c3b2d-c8s9p\" (UID: \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:21.114254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.114116    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b9c3b2d-c8s9p\" (UID: \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:21.114254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.114187    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b9c3b2d-c8s9p\" (UID: \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:21.114806 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.114783    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b9c3b2d-c8s9p\" (UID: \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:21.116899 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.116855    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b9c3b2d-c8s9p\" (UID: \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:21.120030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.120005    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"]
Nov 02 07:50:21.137927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.137872    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hl6\" (UniqueName: \"kubernetes.io/projected/d71db5b5-4f54-43ab-ba8d-05e114104d7d-kube-api-access-85hl6\") pod \"maas-api-revoke-sub-4b9c3b2d-c8s9p\" (UID: \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:21.156123 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.156098    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"]
Nov 02 07:50:21.238534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.238455    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:21.602742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:21.602704    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"]
Nov 02 07:50:21.603232 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:50:21.603195    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71db5b5_4f54_43ab_ba8d_05e114104d7d.slice/crio-88ec425abd10d1dbef9bb2a5b4c1d95e4579c4be5dfaf969b9c2ddd5e1c4caff WatchSource:0}: Error finding container 88ec425abd10d1dbef9bb2a5b4c1d95e4579c4be5dfaf969b9c2ddd5e1c4caff: Status 404 returned error can't find the container with id 88ec425abd10d1dbef9bb2a5b4c1d95e4579c4be5dfaf969b9c2ddd5e1c4caff
Nov 02 07:50:22.083481 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.083445    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:50:22.244023 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.243935    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-data\") pod \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") "
Nov 02 07:50:22.244023 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.243994    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-credential-socket\") pod \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") "
Nov 02 07:50:22.244242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.244048    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-socket\") pod \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") "
Nov 02 07:50:22.244242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.244140    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-podinfo\") pod \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") "
Nov 02 07:50:22.244242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.244177    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-kube-api-access-pvrnk\" (UniqueName: \"kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-kube-api-access-pvrnk\") pod \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") "
Nov 02 07:50:22.244242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.244229    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-envoy\") pod \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") "
Nov 02 07:50:22.244242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.244202    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-data" pod "5c749a89-2eb7-4bdf-810a-ffae61b7e264" (UID: "5c749a89-2eb7-4bdf-810a-ffae61b7e264"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:22.244514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.244277    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-credential-socket" pod "5c749a89-2eb7-4bdf-810a-ffae61b7e264" (UID: "5c749a89-2eb7-4bdf-810a-ffae61b7e264"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:22.244514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.244305    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-token\" (UniqueName: \"kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-token\") pod \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") "
Nov 02 07:50:22.244514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.244360    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-certs\") pod \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") "
Nov 02 07:50:22.244514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.244371    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-socket" pod "5c749a89-2eb7-4bdf-810a-ffae61b7e264" (UID: "5c749a89-2eb7-4bdf-810a-ffae61b7e264"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:22.244514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.244420    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istiod-ca-cert\") pod \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\" (UID: \"5c749a89-2eb7-4bdf-810a-ffae61b7e264\") "
Nov 02 07:50:22.244776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.244665    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-certs" pod "5c749a89-2eb7-4bdf-810a-ffae61b7e264" (UID: "5c749a89-2eb7-4bdf-810a-ffae61b7e264"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:22.244904 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.244865    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istiod-ca-cert" pod "5c749a89-2eb7-4bdf-810a-ffae61b7e264" (UID: "5c749a89-2eb7-4bdf-810a-ffae61b7e264"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:22.245101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.245079    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:22.245171 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.245106    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:22.245171 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.245121    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:22.245171 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.245134    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:22.245171 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.245147    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:22.246523 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.246499    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-token" pod "5c749a89-2eb7-4bdf-810a-ffae61b7e264" (UID: "5c749a89-2eb7-4bdf-810a-ffae61b7e264"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:22.246652 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.246604    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-podinfo" pod "5c749a89-2eb7-4bdf-810a-ffae61b7e264" (UID: "5c749a89-2eb7-4bdf-810a-ffae61b7e264"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:50:22.246926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.246875    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-envoy" pod "5c749a89-2eb7-4bdf-810a-ffae61b7e264" (UID: "5c749a89-2eb7-4bdf-810a-ffae61b7e264"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:22.247004 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.246938    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-kube-api-access-pvrnk" pod "5c749a89-2eb7-4bdf-810a-ffae61b7e264" (UID: "5c749a89-2eb7-4bdf-810a-ffae61b7e264"). InnerVolumeSpecName "kube-api-access-pvrnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:22.346991 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.346884    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:22.346991 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.346996    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvrnk\" (UniqueName: \"kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-kube-api-access-pvrnk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:22.347228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.347012    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:22.347228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.347024    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c749a89-2eb7-4bdf-810a-ffae61b7e264-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:22.494397 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.494313    4898 generic.go:356] "Generic (PLEG): container finished" podID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerID="409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a" exitCode=0
Nov 02 07:50:22.494747 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.494388    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"
Nov 02 07:50:22.494824 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.494390    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" event={"ID":"5c749a89-2eb7-4bdf-810a-ffae61b7e264","Type":"ContainerDied","Data":"409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a"}
Nov 02 07:50:22.494915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.494878    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc" event={"ID":"5c749a89-2eb7-4bdf-810a-ffae61b7e264","Type":"ContainerDied","Data":"1baa1af1ab1651f3231b8d25f95be05150c3bbf787b389f728ed12ec639fd279"}
Nov 02 07:50:22.494994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.494929    4898 scope.go:122] "RemoveContainer" containerID="409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a"
Nov 02 07:50:22.496367 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.496341    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p" event={"ID":"d71db5b5-4f54-43ab-ba8d-05e114104d7d","Type":"ContainerStarted","Data":"f7bd7aa017d08e4d69988e57983bbfb568b20f7cd6461b0408f621364da5d3fc"}
Nov 02 07:50:22.496468 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.496388    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p" event={"ID":"d71db5b5-4f54-43ab-ba8d-05e114104d7d","Type":"ContainerStarted","Data":"88ec425abd10d1dbef9bb2a5b4c1d95e4579c4be5dfaf969b9c2ddd5e1c4caff"}
Nov 02 07:50:22.507612 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.507591    4898 scope.go:122] "RemoveContainer" containerID="612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea"
Nov 02 07:50:22.517900 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.517868    4898 scope.go:122] "RemoveContainer" containerID="409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a"
Nov 02 07:50:22.518162 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:50:22.518144    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a\": container with ID starting with 409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a not found: ID does not exist" containerID="409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a"
Nov 02 07:50:22.518214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.518172    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a"} err="failed to get container status \"409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a\": rpc error: code = NotFound desc = could not find container \"409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a\": container with ID starting with 409305afca6c9c5a7dfd438b655a07ad4a3e3b359856567a1ca5782e545ba76a not found: ID does not exist"
Nov 02 07:50:22.518214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.518190    4898 scope.go:122] "RemoveContainer" containerID="612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea"
Nov 02 07:50:22.518427 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:50:22.518413    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea\": container with ID starting with 612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea not found: ID does not exist" containerID="612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea"
Nov 02 07:50:22.518467 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.518430    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea"} err="failed to get container status \"612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea\": rpc error: code = NotFound desc = could not find container \"612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea\": container with ID starting with 612577d2110228d9e9dddfb858068f5c84e75639570e6acf460f7629136e34ea not found: ID does not exist"
Nov 02 07:50:22.552908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.552843    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p" podStartSLOduration=2.552824948 podStartE2EDuration="2.552824948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:22.550598894 +0000 UTC m=+1573.240925864" watchObservedRunningTime="2026-11-02 07:50:22.552824948 +0000 UTC m=+1573.243151915"
Nov 02 07:50:22.589971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.589942    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"]
Nov 02 07:50:22.601688 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:22.601661    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b2f76f-openshift-default-74dfd8679c-sr5kc"]
Nov 02 07:50:23.469483 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.469437    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" path="/var/lib/kubelet/pods/5c749a89-2eb7-4bdf-810a-ffae61b7e264/volumes"
Nov 02 07:50:23.509872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.509830    4898 generic.go:356] "Generic (PLEG): container finished" podID="d71db5b5-4f54-43ab-ba8d-05e114104d7d" containerID="f7bd7aa017d08e4d69988e57983bbfb568b20f7cd6461b0408f621364da5d3fc" exitCode=0
Nov 02 07:50:23.510078 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.509938    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p" event={"ID":"d71db5b5-4f54-43ab-ba8d-05e114104d7d","Type":"ContainerDied","Data":"f7bd7aa017d08e4d69988e57983bbfb568b20f7cd6461b0408f621364da5d3fc"}
Nov 02 07:50:23.574222 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.574189    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"]
Nov 02 07:50:23.575180 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.575158    4898 cpu_manager.go:414] "containerMap: removing container" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerName="istio-proxy"
Nov 02 07:50:23.575239 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.575183    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerName="istio-proxy"
Nov 02 07:50:23.575408 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.575395    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerName="istio-proxy"
Nov 02 07:50:23.575454 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.575424    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerName="istio-proxy"
Nov 02 07:50:23.584383 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.584353    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.591393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.591365    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"]
Nov 02 07:50:23.596944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.592337    4898 cpu_manager.go:414] "containerMap: removing container" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerName="istio-proxy"
Nov 02 07:50:23.596944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.592362    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c749a89-2eb7-4bdf-810a-ffae61b7e264" containerName="istio-proxy"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.602106    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.602511    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"]
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.605289    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smw2n\" (UniqueName: \"kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-kube-api-access-smw2n\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.605356    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.605404    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.605429    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.605478    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.605808    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.605838    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1409db03-535a-4343-902e-bcfcadd91650-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.605862    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1409db03-535a-4343-902e-bcfcadd91650-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.605932    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.605964    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.606023    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52wgq\" (UniqueName: \"kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-kube-api-access-52wgq\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.606050    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.606078    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.606115    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1127b479-5f27-43d2-a86b-069a7cd84d39-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.606157    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.606184    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.606260    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.612985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.606355    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.623876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.623845    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"]
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707323    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smw2n\" (UniqueName: \"kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-kube-api-access-smw2n\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707390    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707419    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707446    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707491    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707542    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707582    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1409db03-535a-4343-902e-bcfcadd91650-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707607    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1409db03-535a-4343-902e-bcfcadd91650-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707641    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707673    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707727    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52wgq\" (UniqueName: \"kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-kube-api-access-52wgq\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707752    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707780    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707816    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1127b479-5f27-43d2-a86b-069a7cd84d39-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707857    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.707911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707883    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.708937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.707972    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.708937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.708053    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.708937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.708882    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.709732 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.709704    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.713074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.710344    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.713265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.710785    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.715769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.711034    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.715769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.711248    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.715769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.711752    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.715769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.712446    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1127b479-5f27-43d2-a86b-069a7cd84d39-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.715769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.712611    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1409db03-535a-4343-902e-bcfcadd91650-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.715769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.712672    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.729940 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.718407    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.729940 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.723796    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.733256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.731010    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.733256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.733186    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smw2n\" (UniqueName: \"kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-kube-api-access-smw2n\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.733872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.733787    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1409db03-535a-4343-902e-bcfcadd91650-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.734946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.734645    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52wgq\" (UniqueName: \"kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-kube-api-access-52wgq\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.735698 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.735672    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:23.736518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.736494    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.906468 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.906197    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:23.949510 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:23.949310    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:24.248785 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.248705    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"]
Nov 02 07:50:24.255061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.255031    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:24.325528 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.325433    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"]
Nov 02 07:50:24.395822 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:50:24.395722    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1409db03_535a_4343_902e_bcfcadd91650.slice/crio-c897d66323875730fa166ed82beea530362ee3b6ddf7137bcd39234d605a1aa6 WatchSource:0}: Error finding container c897d66323875730fa166ed82beea530362ee3b6ddf7137bcd39234d605a1aa6: Status 404 returned error can't find the container with id c897d66323875730fa166ed82beea530362ee3b6ddf7137bcd39234d605a1aa6
Nov 02 07:50:24.402179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.402116    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:24.402455 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.402412    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:24.402645 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.402623    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:24.416060 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:50:24.415971    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1127b479_5f27_43d2_a86b_069a7cd84d39.slice/crio-3cd7dcb1f1095e825ecc1be8f3ce0a37efbd4244b7b37afeeedf5260c70148be WatchSource:0}: Error finding container 3cd7dcb1f1095e825ecc1be8f3ce0a37efbd4244b7b37afeeedf5260c70148be: Status 404 returned error can't find the container with id 3cd7dcb1f1095e825ecc1be8f3ce0a37efbd4244b7b37afeeedf5260c70148be
Nov 02 07:50:24.432147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.432004    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"]
Nov 02 07:50:24.432147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.432115    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"]
Nov 02 07:50:24.438663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.437955    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c590519-4cvjz\" (UID: \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:24.438663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.438038    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncqkb\" (UniqueName: \"kubernetes.io/projected/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-kube-api-access-ncqkb\") pod \"maas-api-revoke-sub-0c590519-4cvjz\" (UID: \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:24.438663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.438098    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c590519-4cvjz\" (UID: \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:24.464544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.464495    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:24.478174 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.478103    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:24.478644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.478281    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:50:24.522428 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.522393    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4" event={"ID":"1409db03-535a-4343-902e-bcfcadd91650","Type":"ContainerStarted","Data":"c897d66323875730fa166ed82beea530362ee3b6ddf7137bcd39234d605a1aa6"}
Nov 02 07:50:24.525096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.525068    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5" event={"ID":"1127b479-5f27-43d2-a86b-069a7cd84d39","Type":"ContainerStarted","Data":"3cd7dcb1f1095e825ecc1be8f3ce0a37efbd4244b7b37afeeedf5260c70148be"}
Nov 02 07:50:24.539651 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.539625    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c590519-4cvjz\" (UID: \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:24.539832 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.539813    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncqkb\" (UniqueName: \"kubernetes.io/projected/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-kube-api-access-ncqkb\") pod \"maas-api-revoke-sub-0c590519-4cvjz\" (UID: \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:24.540021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.539991    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c590519-4cvjz\" (UID: \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:24.542727 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.542702    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c590519-4cvjz\" (UID: \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:24.543592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.543572    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c590519-4cvjz\" (UID: \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:24.571111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.571052    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncqkb\" (UniqueName: \"kubernetes.io/projected/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-kube-api-access-ncqkb\") pod \"maas-api-revoke-sub-0c590519-4cvjz\" (UID: \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:24.650204 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.643814    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:24.879632 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:24.879146    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:25.061948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.061909    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71db5b5-4f54-43ab-ba8d-05e114104d7d-kube-api-access-85hl6\" (UniqueName: \"kubernetes.io/projected/d71db5b5-4f54-43ab-ba8d-05e114104d7d-kube-api-access-85hl6\") pod \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\" (UID: \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\") "
Nov 02 07:50:25.062129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.061963    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-openshift-ca\") pod \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\" (UID: \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\") "
Nov 02 07:50:25.062129 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.062038    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-xks-ca\") pod \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\" (UID: \"d71db5b5-4f54-43ab-ba8d-05e114104d7d\") "
Nov 02 07:50:25.063016 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.062988    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-openshift-ca" pod "d71db5b5-4f54-43ab-ba8d-05e114104d7d" (UID: "d71db5b5-4f54-43ab-ba8d-05e114104d7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:25.063726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.063701    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:25.064710 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.064680    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71db5b5-4f54-43ab-ba8d-05e114104d7d-kube-api-access-85hl6" pod "d71db5b5-4f54-43ab-ba8d-05e114104d7d" (UID: "d71db5b5-4f54-43ab-ba8d-05e114104d7d"). InnerVolumeSpecName "kube-api-access-85hl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:25.065453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.065417    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-xks-ca" pod "d71db5b5-4f54-43ab-ba8d-05e114104d7d" (UID: "d71db5b5-4f54-43ab-ba8d-05e114104d7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:25.148255 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:50:25.148207    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3e3f0a_c2f3_4394_959c_3d8cc3ecb77f.slice/crio-4ca135f833cf6c6c573d17c499e8326182aaf28d9ff6e6a8147c321b8c1012da WatchSource:0}: Error finding container 4ca135f833cf6c6c573d17c499e8326182aaf28d9ff6e6a8147c321b8c1012da: Status 404 returned error can't find the container with id 4ca135f833cf6c6c573d17c499e8326182aaf28d9ff6e6a8147c321b8c1012da
Nov 02 07:50:25.150836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.150803    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"]
Nov 02 07:50:25.166876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.166841    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71db5b5-4f54-43ab-ba8d-05e114104d7d-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:25.166876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.166877    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85hl6\" (UniqueName: \"kubernetes.io/projected/d71db5b5-4f54-43ab-ba8d-05e114104d7d-kube-api-access-85hl6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:25.586925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.582104    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p" event={"ID":"d71db5b5-4f54-43ab-ba8d-05e114104d7d","Type":"ContainerDied","Data":"88ec425abd10d1dbef9bb2a5b4c1d95e4579c4be5dfaf969b9c2ddd5e1c4caff"}
Nov 02 07:50:25.586925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.582169    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88ec425abd10d1dbef9bb2a5b4c1d95e4579c4be5dfaf969b9c2ddd5e1c4caff"
Nov 02 07:50:25.586925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.582249    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"
Nov 02 07:50:25.596844 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.594960    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4" event={"ID":"1409db03-535a-4343-902e-bcfcadd91650","Type":"ContainerStarted","Data":"819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b"}
Nov 02 07:50:25.624280 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.624238    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5" event={"ID":"1127b479-5f27-43d2-a86b-069a7cd84d39","Type":"ContainerStarted","Data":"ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97"}
Nov 02 07:50:25.628301 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.628263    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz" event={"ID":"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f","Type":"ContainerStarted","Data":"4ca135f833cf6c6c573d17c499e8326182aaf28d9ff6e6a8147c321b8c1012da"}
Nov 02 07:50:25.760174 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.751215    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5" podStartSLOduration=2.751191507 podStartE2EDuration="2.751191507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:25.712329954 +0000 UTC m=+1576.402656919" watchObservedRunningTime="2026-11-02 07:50:25.751191507 +0000 UTC m=+1576.441518471"
Nov 02 07:50:25.760174 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.752651    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4" podStartSLOduration=2.752638567 podStartE2EDuration="2.752638567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:25.654480523 +0000 UTC m=+1576.344807489" watchObservedRunningTime="2026-11-02 07:50:25.752638567 +0000 UTC m=+1576.442965534"
Nov 02 07:50:25.910915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.910446    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:25.910915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.910672    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 07:50:25.910915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.910738    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4" podUID="1409db03-535a-4343-902e-bcfcadd91650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 07:50:25.951801 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.951501    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 02 07:50:25.951801 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.951552    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5" podUID="1127b479-5f27-43d2-a86b-069a7cd84d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 02 07:50:25.951801 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:25.951749    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:26.123087 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.122972    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh" podUID="1f7b4fa4-1a09-46cb-97e3-f9feed6ab765" containerName="payload-processing" containerID="cri-o://d6c8568b2c596c1a1bf0165b964dafc69e9e4593014c23aa4227e2e282b9de19" gracePeriod=25
Nov 02 07:50:26.160912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.158961    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm" podUID="8a79a41d-1043-4601-ab0b-ee16cbd02f9d" containerName="payload-pre-processing" containerID="cri-o://b450509ed37d972df3319a1124a81276989ea353dd70644195e9765cd2058a75" gracePeriod=25
Nov 02 07:50:26.733808 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.733764    4898 generic.go:356] "Generic (PLEG): container finished" podID="8a79a41d-1043-4601-ab0b-ee16cbd02f9d" containerID="b450509ed37d972df3319a1124a81276989ea353dd70644195e9765cd2058a75" exitCode=0
Nov 02 07:50:26.734587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.734082    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm" event={"ID":"8a79a41d-1043-4601-ab0b-ee16cbd02f9d","Type":"ContainerDied","Data":"b450509ed37d972df3319a1124a81276989ea353dd70644195e9765cd2058a75"}
Nov 02 07:50:26.736761 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.736609    4898 generic.go:356] "Generic (PLEG): container finished" podID="1f7b4fa4-1a09-46cb-97e3-f9feed6ab765" containerID="d6c8568b2c596c1a1bf0165b964dafc69e9e4593014c23aa4227e2e282b9de19" exitCode=0
Nov 02 07:50:26.736761 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.736692    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh" event={"ID":"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765","Type":"ContainerDied","Data":"d6c8568b2c596c1a1bf0165b964dafc69e9e4593014c23aa4227e2e282b9de19"}
Nov 02 07:50:26.741867 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.740648    4898 generic.go:356] "Generic (PLEG): container finished" podID="ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f" containerID="871c378caba3e82e58608a86840b4fcea561236f21195a3101a6685d41fbf166" exitCode=0
Nov 02 07:50:26.741867 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.740983    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz" event={"ID":"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f","Type":"ContainerDied","Data":"871c378caba3e82e58608a86840b4fcea561236f21195a3101a6685d41fbf166"}
Nov 02 07:50:26.913935 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.908559    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 07:50:26.913935 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.908617    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4" podUID="1409db03-535a-4343-902e-bcfcadd91650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 07:50:26.940909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.940868    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"
Nov 02 07:50:26.951977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.951942    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 02 07:50:26.952257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:26.952226    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5" podUID="1127b479-5f27-43d2-a86b-069a7cd84d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 02 07:50:27.050948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.050911    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-kube-api-access-b8dqz\" (UniqueName: \"kubernetes.io/projected/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-kube-api-access-b8dqz\") pod \"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765\" (UID: \"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765\") "
Nov 02 07:50:27.051700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.051677    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-plugins-config-volume\") pod \"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765\" (UID: \"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765\") "
Nov 02 07:50:27.052192 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.052168    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-plugins-config-volume" pod "1f7b4fa4-1a09-46cb-97e3-f9feed6ab765" (UID: "1f7b4fa4-1a09-46cb-97e3-f9feed6ab765"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:27.052714 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.052690    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:27.054300 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.054037    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-kube-api-access-b8dqz" pod "1f7b4fa4-1a09-46cb-97e3-f9feed6ab765" (UID: "1f7b4fa4-1a09-46cb-97e3-f9feed6ab765"). InnerVolumeSpecName "kube-api-access-b8dqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:27.136092 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.136060    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"
Nov 02 07:50:27.153989 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.153960    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8dqz\" (UniqueName: \"kubernetes.io/projected/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765-kube-api-access-b8dqz\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:27.275914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.255200    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-plugins-config-volume\") pod \"8a79a41d-1043-4601-ab0b-ee16cbd02f9d\" (UID: \"8a79a41d-1043-4601-ab0b-ee16cbd02f9d\") "
Nov 02 07:50:27.276858 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.276829    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-kube-api-access-zm4pf\" (UniqueName: \"kubernetes.io/projected/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-kube-api-access-zm4pf\") pod \"8a79a41d-1043-4601-ab0b-ee16cbd02f9d\" (UID: \"8a79a41d-1043-4601-ab0b-ee16cbd02f9d\") "
Nov 02 07:50:27.278091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.276565    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-plugins-config-volume" pod "8a79a41d-1043-4601-ab0b-ee16cbd02f9d" (UID: "8a79a41d-1043-4601-ab0b-ee16cbd02f9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:27.311130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.311068    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-kube-api-access-zm4pf" pod "8a79a41d-1043-4601-ab0b-ee16cbd02f9d" (UID: "8a79a41d-1043-4601-ab0b-ee16cbd02f9d"). InnerVolumeSpecName "kube-api-access-zm4pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:27.396044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.395974    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm4pf\" (UniqueName: \"kubernetes.io/projected/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-kube-api-access-zm4pf\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:27.396044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.396010    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a79a41d-1043-4601-ab0b-ee16cbd02f9d-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:27.749348 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.749321    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"
Nov 02 07:50:27.750516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.749369    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm" event={"ID":"8a79a41d-1043-4601-ab0b-ee16cbd02f9d","Type":"ContainerDied","Data":"6b44ac0add256cf9c23f47d1124c9cdad53f1e22afff014276401feb6fee0dc8"}
Nov 02 07:50:27.750672 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.750541    4898 scope.go:122] "RemoveContainer" containerID="b450509ed37d972df3319a1124a81276989ea353dd70644195e9765cd2058a75"
Nov 02 07:50:27.752584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.752561    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh" event={"ID":"1f7b4fa4-1a09-46cb-97e3-f9feed6ab765","Type":"ContainerDied","Data":"ccf360aecfea29de871ffc0b6c9b98a09548faa00673197f05572c85705cc1d6"}
Nov 02 07:50:27.752749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.752667    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"
Nov 02 07:50:27.824224 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.820355    4898 scope.go:122] "RemoveContainer" containerID="d6c8568b2c596c1a1bf0165b964dafc69e9e4593014c23aa4227e2e282b9de19"
Nov 02 07:50:27.829020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.828960    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"]
Nov 02 07:50:27.836102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.834605    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-677726-6c9ddcb55f-grwgm"]
Nov 02 07:50:27.925995 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.923491    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 07:50:27.925995 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.923570    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4" podUID="1409db03-535a-4343-902e-bcfcadd91650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 07:50:27.926280 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.926035    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"]
Nov 02 07:50:27.926439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.926406    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-677726-699b6cbdf7-nbzbh"]
Nov 02 07:50:27.953777 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.953040    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 02 07:50:27.954006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:27.953980    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5" podUID="1127b479-5f27-43d2-a86b-069a7cd84d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 02 07:50:28.222362 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.222324    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:28.337248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.337209    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-kube-api-access-ncqkb\" (UniqueName: \"kubernetes.io/projected/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-kube-api-access-ncqkb\") pod \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\" (UID: \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\") "
Nov 02 07:50:28.337463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.337316    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-xks-ca\") pod \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\" (UID: \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\") "
Nov 02 07:50:28.337524 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.337496    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-openshift-ca\") pod \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\" (UID: \"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f\") "
Nov 02 07:50:28.338126 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.338086    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-openshift-ca" pod "ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f" (UID: "ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:28.339747 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.339724    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-xks-ca" pod "ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f" (UID: "ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:28.340003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.339979    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-kube-api-access-ncqkb" pod "ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f" (UID: "ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f"). InnerVolumeSpecName "kube-api-access-ncqkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:28.439551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.439520    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:28.439774 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.439760    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncqkb\" (UniqueName: \"kubernetes.io/projected/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-kube-api-access-ncqkb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:28.439842 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.439834    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:28.780951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.780832    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz" event={"ID":"ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f","Type":"ContainerDied","Data":"4ca135f833cf6c6c573d17c499e8326182aaf28d9ff6e6a8147c321b8c1012da"}
Nov 02 07:50:28.781611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.781590    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ca135f833cf6c6c573d17c499e8326182aaf28d9ff6e6a8147c321b8c1012da"
Nov 02 07:50:28.781743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.781529    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"
Nov 02 07:50:28.923600 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.923562    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 07:50:28.923917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.923865    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4" podUID="1409db03-535a-4343-902e-bcfcadd91650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 07:50:28.964491 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:28.964454    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 02 07:50:29.021756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:29.021698    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5" podUID="1127b479-5f27-43d2-a86b-069a7cd84d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 02 07:50:29.519913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:29.518172    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7b4fa4-1a09-46cb-97e3-f9feed6ab765" path="/var/lib/kubelet/pods/1f7b4fa4-1a09-46cb-97e3-f9feed6ab765/volumes"
Nov 02 07:50:29.519913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:29.518843    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a79a41d-1043-4601-ab0b-ee16cbd02f9d" path="/var/lib/kubelet/pods/8a79a41d-1043-4601-ab0b-ee16cbd02f9d/volumes"
Nov 02 07:50:29.918036 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:29.918000    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 07:50:29.918602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:29.918570    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4" podUID="1409db03-535a-4343-902e-bcfcadd91650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 07:50:29.950936 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:29.950910    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 02 07:50:29.951085 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:29.950961    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5" podUID="1127b479-5f27-43d2-a86b-069a7cd84d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 02 07:50:30.908960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:30.908881    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 02 07:50:30.909123 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:30.908987    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4" podUID="1409db03-535a-4343-902e-bcfcadd91650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 02 07:50:30.950220 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:30.950184    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 02 07:50:30.950616 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:30.950249    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5" podUID="1127b479-5f27-43d2-a86b-069a7cd84d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 02 07:50:32.400755 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:32.400688    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:32.403089 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:32.402968    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:32.404069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:32.404047    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:50:32.477854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:32.477820    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:32.478867 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:32.478833    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:32.480271 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:32.480240    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:50:34.672655 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.672616    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"]
Nov 02 07:50:34.673305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.673287    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d71db5b5-4f54-43ab-ba8d-05e114104d7d" containerName="revoke-keys"
Nov 02 07:50:34.673358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.673307    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71db5b5-4f54-43ab-ba8d-05e114104d7d" containerName="revoke-keys"
Nov 02 07:50:34.673358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.673320    4898 cpu_manager.go:414] "containerMap: removing container" podUID="8a79a41d-1043-4601-ab0b-ee16cbd02f9d" containerName="payload-pre-processing"
Nov 02 07:50:34.673358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.673326    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a79a41d-1043-4601-ab0b-ee16cbd02f9d" containerName="payload-pre-processing"
Nov 02 07:50:34.673358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.673351    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1f7b4fa4-1a09-46cb-97e3-f9feed6ab765" containerName="payload-processing"
Nov 02 07:50:34.673358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.673356    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7b4fa4-1a09-46cb-97e3-f9feed6ab765" containerName="payload-processing"
Nov 02 07:50:34.673521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.673361    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f" containerName="revoke-keys"
Nov 02 07:50:34.673521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.673366    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f" containerName="revoke-keys"
Nov 02 07:50:34.673521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.673467    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7b4fa4-1a09-46cb-97e3-f9feed6ab765" containerName="payload-processing"
Nov 02 07:50:34.673521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.673473    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f" containerName="revoke-keys"
Nov 02 07:50:34.673521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.673487    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71db5b5-4f54-43ab-ba8d-05e114104d7d" containerName="revoke-keys"
Nov 02 07:50:34.673521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.673492    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a79a41d-1043-4601-ab0b-ee16cbd02f9d" containerName="payload-pre-processing"
Nov 02 07:50:34.677147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.677126    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:34.698185 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.698152    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"]
Nov 02 07:50:34.704580 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.704546    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:50:34.704746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.704564    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:50:34.704973 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.704951    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:50:34.707089 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.707062    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:50:34.820818 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.820785    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1936d576-bggck\" (UID: \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:34.820818 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.820824    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1936d576-bggck\" (UID: \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:34.821054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.821010    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-559qq\" (UniqueName: \"kubernetes.io/projected/db0eb42b-9a10-49a3-8ad6-97a7bff68929-kube-api-access-559qq\") pod \"maas-api-revoke-sub-1936d576-bggck\" (UID: \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:34.921983 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.921943    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-559qq\" (UniqueName: \"kubernetes.io/projected/db0eb42b-9a10-49a3-8ad6-97a7bff68929-kube-api-access-559qq\") pod \"maas-api-revoke-sub-1936d576-bggck\" (UID: \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:34.922187 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.922171    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1936d576-bggck\" (UID: \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:34.922245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.922198    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1936d576-bggck\" (UID: \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:34.923097 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.923023    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1936d576-bggck\" (UID: \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:34.924837 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.924812    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1936d576-bggck\" (UID: \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:34.946627 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.946601    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-559qq\" (UniqueName: \"kubernetes.io/projected/db0eb42b-9a10-49a3-8ad6-97a7bff68929-kube-api-access-559qq\") pod \"maas-api-revoke-sub-1936d576-bggck\" (UID: \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:34.997814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:34.997778    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:35.313364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.307777    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"]
Nov 02 07:50:35.316315 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:50:35.316270    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0eb42b_9a10_49a3_8ad6_97a7bff68929.slice/crio-a13e3b57c7951568a8d0469f679778e17bac76ef1173942d62368ed6e4f68499 WatchSource:0}: Error finding container a13e3b57c7951568a8d0469f679778e17bac76ef1173942d62368ed6e4f68499: Status 404 returned error can't find the container with id a13e3b57c7951568a8d0469f679778e17bac76ef1173942d62368ed6e4f68499
Nov 02 07:50:35.659263 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.659233    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"]
Nov 02 07:50:35.667435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.667401    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"]
Nov 02 07:50:35.667721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.667607    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv" podUID="7ccd53d3-0f42-423c-82bd-642e4de884a3" containerName="payload-processing" containerID="cri-o://75267ce18581210c00cd48bfb40cc4f391b687bed55bb86c9f79a391e50390c4" gracePeriod=30
Nov 02 07:50:35.668141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.667992    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:35.695415 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.695377    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"]
Nov 02 07:50:35.734010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.733740    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn5qd\" (UniqueName: \"kubernetes.io/projected/fbbceb31-c5f2-4b7a-8917-adf105c31df0-kube-api-access-pn5qd\") pod \"maas-api-revoke-sub-1dd704e4-zdsts\" (UID: \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:35.734010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.733786    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dd704e4-zdsts\" (UID: \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:35.734010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.733811    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dd704e4-zdsts\" (UID: \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:35.835289 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.835255    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn5qd\" (UniqueName: \"kubernetes.io/projected/fbbceb31-c5f2-4b7a-8917-adf105c31df0-kube-api-access-pn5qd\") pod \"maas-api-revoke-sub-1dd704e4-zdsts\" (UID: \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:35.835455 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.835307    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dd704e4-zdsts\" (UID: \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:35.835455 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.835345    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dd704e4-zdsts\" (UID: \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:35.836033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.836011    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dd704e4-zdsts\" (UID: \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:35.837708 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.837686    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dd704e4-zdsts\" (UID: \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:35.873589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.873561    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn5qd\" (UniqueName: \"kubernetes.io/projected/fbbceb31-c5f2-4b7a-8917-adf105c31df0-kube-api-access-pn5qd\") pod \"maas-api-revoke-sub-1dd704e4-zdsts\" (UID: \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:35.876729 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.876695    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck" event={"ID":"db0eb42b-9a10-49a3-8ad6-97a7bff68929","Type":"ContainerStarted","Data":"c83fb64c0a9f89c2f5d32dc232b9cc9eb5957b07909f36f584ee4f243fc7874c"}
Nov 02 07:50:35.876866 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.876756    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck" event={"ID":"db0eb42b-9a10-49a3-8ad6-97a7bff68929","Type":"ContainerStarted","Data":"a13e3b57c7951568a8d0469f679778e17bac76ef1173942d62368ed6e4f68499"}
Nov 02 07:50:35.878759 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.878738    4898 generic.go:356] "Generic (PLEG): container finished" podID="7ccd53d3-0f42-423c-82bd-642e4de884a3" containerID="75267ce18581210c00cd48bfb40cc4f391b687bed55bb86c9f79a391e50390c4" exitCode=0
Nov 02 07:50:35.878856 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.878815    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv" event={"ID":"7ccd53d3-0f42-423c-82bd-642e4de884a3","Type":"ContainerDied","Data":"75267ce18581210c00cd48bfb40cc4f391b687bed55bb86c9f79a391e50390c4"}
Nov 02 07:50:35.934319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.931958    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck" podStartSLOduration=1.931936598 podStartE2EDuration="1.931936598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:35.93004759 +0000 UTC m=+1586.620374553" watchObservedRunningTime="2026-11-02 07:50:35.931936598 +0000 UTC m=+1586.622263561"
Nov 02 07:50:35.971218 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:35.971192    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"
Nov 02 07:50:36.042035 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.040758    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:36.042035 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.041654    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ccd53d3-0f42-423c-82bd-642e4de884a3-kube-api-access-c9jm9\" (UniqueName: \"kubernetes.io/projected/7ccd53d3-0f42-423c-82bd-642e4de884a3-kube-api-access-c9jm9\") pod \"7ccd53d3-0f42-423c-82bd-642e4de884a3\" (UID: \"7ccd53d3-0f42-423c-82bd-642e4de884a3\") "
Nov 02 07:50:36.042035 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.041728    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ccd53d3-0f42-423c-82bd-642e4de884a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ccd53d3-0f42-423c-82bd-642e4de884a3-plugins-config-volume\") pod \"7ccd53d3-0f42-423c-82bd-642e4de884a3\" (UID: \"7ccd53d3-0f42-423c-82bd-642e4de884a3\") "
Nov 02 07:50:36.072927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.042922    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ccd53d3-0f42-423c-82bd-642e4de884a3-plugins-config-volume" pod "7ccd53d3-0f42-423c-82bd-642e4de884a3" (UID: "7ccd53d3-0f42-423c-82bd-642e4de884a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:36.145924 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.144864    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ccd53d3-0f42-423c-82bd-642e4de884a3-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:36.177912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.171519    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ccd53d3-0f42-423c-82bd-642e4de884a3-kube-api-access-c9jm9" pod "7ccd53d3-0f42-423c-82bd-642e4de884a3" (UID: "7ccd53d3-0f42-423c-82bd-642e4de884a3"). InnerVolumeSpecName "kube-api-access-c9jm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:36.246209 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.246172    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9jm9\" (UniqueName: \"kubernetes.io/projected/7ccd53d3-0f42-423c-82bd-642e4de884a3-kube-api-access-c9jm9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:36.542600 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:50:36.542553    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbceb31_c5f2_4b7a_8917_adf105c31df0.slice/crio-2ad24836d6d4200c96271e69df75d5429e67b774a9137e95b0d2ddaf3836e523 WatchSource:0}: Error finding container 2ad24836d6d4200c96271e69df75d5429e67b774a9137e95b0d2ddaf3836e523: Status 404 returned error can't find the container with id 2ad24836d6d4200c96271e69df75d5429e67b774a9137e95b0d2ddaf3836e523
Nov 02 07:50:36.550530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.550497    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"]
Nov 02 07:50:36.887899 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.887853    4898 generic.go:356] "Generic (PLEG): container finished" podID="db0eb42b-9a10-49a3-8ad6-97a7bff68929" containerID="c83fb64c0a9f89c2f5d32dc232b9cc9eb5957b07909f36f584ee4f243fc7874c" exitCode=0
Nov 02 07:50:36.888309 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.887924    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck" event={"ID":"db0eb42b-9a10-49a3-8ad6-97a7bff68929","Type":"ContainerDied","Data":"c83fb64c0a9f89c2f5d32dc232b9cc9eb5957b07909f36f584ee4f243fc7874c"}
Nov 02 07:50:36.889972 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.889942    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts" event={"ID":"fbbceb31-c5f2-4b7a-8917-adf105c31df0","Type":"ContainerStarted","Data":"4a5c7db07060b64a2843a8cdcb1ac3aa06eafc3f035e23957468332647d0a34c"}
Nov 02 07:50:36.890094 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.889997    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts" event={"ID":"fbbceb31-c5f2-4b7a-8917-adf105c31df0","Type":"ContainerStarted","Data":"2ad24836d6d4200c96271e69df75d5429e67b774a9137e95b0d2ddaf3836e523"}
Nov 02 07:50:36.891451 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.891427    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv" event={"ID":"7ccd53d3-0f42-423c-82bd-642e4de884a3","Type":"ContainerDied","Data":"41a4e5d854dd66c8591a24a4d698f346a34ebfcf53627cbd9e7a92928c675d53"}
Nov 02 07:50:36.891549 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.891472    4898 scope.go:122] "RemoveContainer" containerID="75267ce18581210c00cd48bfb40cc4f391b687bed55bb86c9f79a391e50390c4"
Nov 02 07:50:36.891549 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.891429    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"
Nov 02 07:50:36.975762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:36.975711    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts" podStartSLOduration=1.975697615 podStartE2EDuration="1.975697615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:36.97342161 +0000 UTC m=+1587.663748586" watchObservedRunningTime="2026-11-02 07:50:36.975697615 +0000 UTC m=+1587.666024578"
Nov 02 07:50:37.014259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:37.014199    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"]
Nov 02 07:50:37.019987 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:37.019957    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f3716-6448f975db-4ftsv"]
Nov 02 07:50:37.467474 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:37.467436    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ccd53d3-0f42-423c-82bd-642e4de884a3" path="/var/lib/kubelet/pods/7ccd53d3-0f42-423c-82bd-642e4de884a3/volumes"
Nov 02 07:50:37.901259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:37.901227    4898 generic.go:356] "Generic (PLEG): container finished" podID="fbbceb31-c5f2-4b7a-8917-adf105c31df0" containerID="4a5c7db07060b64a2843a8cdcb1ac3aa06eafc3f035e23957468332647d0a34c" exitCode=0
Nov 02 07:50:37.901708 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:37.901311    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts" event={"ID":"fbbceb31-c5f2-4b7a-8917-adf105c31df0","Type":"ContainerDied","Data":"4a5c7db07060b64a2843a8cdcb1ac3aa06eafc3f035e23957468332647d0a34c"}
Nov 02 07:50:38.090154 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.090126    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:38.180448 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.180354    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0eb42b-9a10-49a3-8ad6-97a7bff68929-kube-api-access-559qq\" (UniqueName: \"kubernetes.io/projected/db0eb42b-9a10-49a3-8ad6-97a7bff68929-kube-api-access-559qq\") pod \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\" (UID: \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\") "
Nov 02 07:50:38.180630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.180573    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-openshift-ca\") pod \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\" (UID: \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\") "
Nov 02 07:50:38.180700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.180632    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-xks-ca\") pod \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\" (UID: \"db0eb42b-9a10-49a3-8ad6-97a7bff68929\") "
Nov 02 07:50:38.181313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.181281    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-openshift-ca" pod "db0eb42b-9a10-49a3-8ad6-97a7bff68929" (UID: "db0eb42b-9a10-49a3-8ad6-97a7bff68929"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:38.183390 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.183358    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-xks-ca" pod "db0eb42b-9a10-49a3-8ad6-97a7bff68929" (UID: "db0eb42b-9a10-49a3-8ad6-97a7bff68929"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:38.183498 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.183364    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0eb42b-9a10-49a3-8ad6-97a7bff68929-kube-api-access-559qq" pod "db0eb42b-9a10-49a3-8ad6-97a7bff68929" (UID: "db0eb42b-9a10-49a3-8ad6-97a7bff68929"). InnerVolumeSpecName "kube-api-access-559qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:38.282575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.282537    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:38.282575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.282569    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0eb42b-9a10-49a3-8ad6-97a7bff68929-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:38.282575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.282578    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-559qq\" (UniqueName: \"kubernetes.io/projected/db0eb42b-9a10-49a3-8ad6-97a7bff68929-kube-api-access-559qq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:38.911296 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.911264    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"
Nov 02 07:50:38.911816 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.911300    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck" event={"ID":"db0eb42b-9a10-49a3-8ad6-97a7bff68929","Type":"ContainerDied","Data":"a13e3b57c7951568a8d0469f679778e17bac76ef1173942d62368ed6e4f68499"}
Nov 02 07:50:38.911816 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:38.911349    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a13e3b57c7951568a8d0469f679778e17bac76ef1173942d62368ed6e4f68499"
Nov 02 07:50:39.083113 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.083082    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:39.201606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.201510    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbceb31-c5f2-4b7a-8917-adf105c31df0-kube-api-access-pn5qd\" (UniqueName: \"kubernetes.io/projected/fbbceb31-c5f2-4b7a-8917-adf105c31df0-kube-api-access-pn5qd\") pod \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\" (UID: \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\") "
Nov 02 07:50:39.201790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.201618    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-xks-ca\") pod \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\" (UID: \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\") "
Nov 02 07:50:39.201790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.201662    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-openshift-ca\") pod \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\" (UID: \"fbbceb31-c5f2-4b7a-8917-adf105c31df0\") "
Nov 02 07:50:39.202075 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.202045    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-openshift-ca" pod "fbbceb31-c5f2-4b7a-8917-adf105c31df0" (UID: "fbbceb31-c5f2-4b7a-8917-adf105c31df0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:39.202412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.202391    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:39.203792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.203763    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-xks-ca" pod "fbbceb31-c5f2-4b7a-8917-adf105c31df0" (UID: "fbbceb31-c5f2-4b7a-8917-adf105c31df0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:39.203792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.203779    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbceb31-c5f2-4b7a-8917-adf105c31df0-kube-api-access-pn5qd" pod "fbbceb31-c5f2-4b7a-8917-adf105c31df0" (UID: "fbbceb31-c5f2-4b7a-8917-adf105c31df0"). InnerVolumeSpecName "kube-api-access-pn5qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:39.303313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.303278    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn5qd\" (UniqueName: \"kubernetes.io/projected/fbbceb31-c5f2-4b7a-8917-adf105c31df0-kube-api-access-pn5qd\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:39.303313 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.303307    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbceb31-c5f2-4b7a-8917-adf105c31df0-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:39.921332 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.921303    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"
Nov 02 07:50:39.921750 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.921344    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts" event={"ID":"fbbceb31-c5f2-4b7a-8917-adf105c31df0","Type":"ContainerDied","Data":"2ad24836d6d4200c96271e69df75d5429e67b774a9137e95b0d2ddaf3836e523"}
Nov 02 07:50:39.921750 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:39.921413    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ad24836d6d4200c96271e69df75d5429e67b774a9137e95b0d2ddaf3836e523"
Nov 02 07:50:42.209556 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.209519    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"]
Nov 02 07:50:42.210214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.210196    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7ccd53d3-0f42-423c-82bd-642e4de884a3" containerName="payload-processing"
Nov 02 07:50:42.210263 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.210216    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ccd53d3-0f42-423c-82bd-642e4de884a3" containerName="payload-processing"
Nov 02 07:50:42.210263 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.210242    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fbbceb31-c5f2-4b7a-8917-adf105c31df0" containerName="revoke-keys"
Nov 02 07:50:42.210263 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.210247    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbceb31-c5f2-4b7a-8917-adf105c31df0" containerName="revoke-keys"
Nov 02 07:50:42.210397 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.210260    4898 cpu_manager.go:414] "containerMap: removing container" podUID="db0eb42b-9a10-49a3-8ad6-97a7bff68929" containerName="revoke-keys"
Nov 02 07:50:42.210397 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.210297    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0eb42b-9a10-49a3-8ad6-97a7bff68929" containerName="revoke-keys"
Nov 02 07:50:42.210949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.210925    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ccd53d3-0f42-423c-82bd-642e4de884a3" containerName="payload-processing"
Nov 02 07:50:42.211058 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.210956    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbceb31-c5f2-4b7a-8917-adf105c31df0" containerName="revoke-keys"
Nov 02 07:50:42.211058 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.210969    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0eb42b-9a10-49a3-8ad6-97a7bff68929" containerName="revoke-keys"
Nov 02 07:50:42.215157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.215138    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"
Nov 02 07:50:42.234683 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.234656    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"]
Nov 02 07:50:42.240786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.240756    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-677726\""
Nov 02 07:50:42.240786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.240756    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-677726-dockercfg-w7llt\""
Nov 02 07:50:42.346168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.346135    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db07204-2fc3-4be1-9730-5f47f311a5ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64\" (UID: \"3db07204-2fc3-4be1-9730-5f47f311a5ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"
Nov 02 07:50:42.346168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.346173    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhtdg\" (UniqueName: \"kubernetes.io/projected/3db07204-2fc3-4be1-9730-5f47f311a5ae-kube-api-access-lhtdg\") pod \"payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64\" (UID: \"3db07204-2fc3-4be1-9730-5f47f311a5ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"
Nov 02 07:50:42.447443 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.447409    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhtdg\" (UniqueName: \"kubernetes.io/projected/3db07204-2fc3-4be1-9730-5f47f311a5ae-kube-api-access-lhtdg\") pod \"payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64\" (UID: \"3db07204-2fc3-4be1-9730-5f47f311a5ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"
Nov 02 07:50:42.447616 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.447604    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db07204-2fc3-4be1-9730-5f47f311a5ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64\" (UID: \"3db07204-2fc3-4be1-9730-5f47f311a5ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"
Nov 02 07:50:42.448364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.448333    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db07204-2fc3-4be1-9730-5f47f311a5ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64\" (UID: \"3db07204-2fc3-4be1-9730-5f47f311a5ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"
Nov 02 07:50:42.471491 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.471424    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhtdg\" (UniqueName: \"kubernetes.io/projected/3db07204-2fc3-4be1-9730-5f47f311a5ae-kube-api-access-lhtdg\") pod \"payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64\" (UID: \"3db07204-2fc3-4be1-9730-5f47f311a5ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"
Nov 02 07:50:42.532766 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.532735    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"
Nov 02 07:50:42.848271 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:50:42.848234    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db07204_2fc3_4be1_9730_5f47f311a5ae.slice/crio-4d92d152b7cda3a6a5b1d62d6bcdb4c4763e6c48c65ee3d9675fa60ef31c928b WatchSource:0}: Error finding container 4d92d152b7cda3a6a5b1d62d6bcdb4c4763e6c48c65ee3d9675fa60ef31c928b: Status 404 returned error can't find the container with id 4d92d152b7cda3a6a5b1d62d6bcdb4c4763e6c48c65ee3d9675fa60ef31c928b
Nov 02 07:50:42.850941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.850704    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"]
Nov 02 07:50:42.974725 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.974681    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64" event={"ID":"3db07204-2fc3-4be1-9730-5f47f311a5ae","Type":"ContainerStarted","Data":"07b7f99d3e245469cc299b9968327f07024831a5bc2d9ee915a33189d96e3f46"}
Nov 02 07:50:42.975155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.974760    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64" event={"ID":"3db07204-2fc3-4be1-9730-5f47f311a5ae","Type":"ContainerStarted","Data":"4d92d152b7cda3a6a5b1d62d6bcdb4c4763e6c48c65ee3d9675fa60ef31c928b"}
Nov 02 07:50:42.975155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:42.975128    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"
Nov 02 07:50:43.025645 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:43.025590    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64" podStartSLOduration=1.02557188 podStartE2EDuration="1.02557188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:43.021081111 +0000 UTC m=+1593.711408075" watchObservedRunningTime="2026-11-02 07:50:43.02557188 +0000 UTC m=+1593.715898892"
Nov 02 07:50:43.773906 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:43.773859    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"]
Nov 02 07:50:43.774357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:43.774088    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" podUID="42283595-1739-4f03-845c-41f9ef5f5073" containerName="istio-proxy" containerID="cri-o://bf78c593adae2369c199bdc0b6a968ed6128774808343cc3a379bd382e34c17a" gracePeriod=30
Nov 02 07:50:44.990609 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:50:44.990572    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:50:46.220823 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:46.220789    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"]
Nov 02 07:50:46.221301 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:46.220997    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" podUID="53de2d77-31bc-415c-8c29-d768438375af" containerName="istio-proxy" containerID="cri-o://812e1dab2a4d367460238b96357e1ec9af08e4f88bbb93298a73c4157bff655b" gracePeriod=30
Nov 02 07:50:47.471126 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:47.471095    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"]
Nov 02 07:50:47.471622 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:47.471290    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerName="istio-proxy" containerID="cri-o://492134a347ade079655780fde0d851f7ce36f495a050eb533c2ec6930e3f6f6d" gracePeriod=30
Nov 02 07:50:47.882936 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:47.882849    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"]
Nov 02 07:50:47.889766 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:47.889737    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:47.906765 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:47.906733    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:50:47.906949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:47.906784    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:50:47.908224 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:47.908078    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"]
Nov 02 07:50:47.918020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:47.917996    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:50:47.918142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:47.918048    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:50:48.042796 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.042755    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"]
Nov 02 07:50:48.043433 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.043408    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88591333-4k9bc\" (UID: \"d0729dd4-778e-4dcd-af53-be08c31754fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:48.043592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.043570    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzs5b\" (UniqueName: \"kubernetes.io/projected/d0729dd4-778e-4dcd-af53-be08c31754fa-kube-api-access-gzs5b\") pod \"maas-api-revoke-sub-88591333-4k9bc\" (UID: \"d0729dd4-778e-4dcd-af53-be08c31754fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:48.043654 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.043609    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88591333-4k9bc\" (UID: \"d0729dd4-778e-4dcd-af53-be08c31754fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:48.047417 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.047396    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:48.071810 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.071780    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"]
Nov 02 07:50:48.144832 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.144744    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5069c276-tjl44\" (UID: \"b92ad256-0ca7-4282-9991-a67bcc91ca18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:48.145035 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.144898    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzs5b\" (UniqueName: \"kubernetes.io/projected/d0729dd4-778e-4dcd-af53-be08c31754fa-kube-api-access-gzs5b\") pod \"maas-api-revoke-sub-88591333-4k9bc\" (UID: \"d0729dd4-778e-4dcd-af53-be08c31754fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:48.145035 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.144953    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88591333-4k9bc\" (UID: \"d0729dd4-778e-4dcd-af53-be08c31754fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:48.145035 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.145012    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5069c276-tjl44\" (UID: \"b92ad256-0ca7-4282-9991-a67bcc91ca18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:48.145221 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.145093    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn6cq\" (UniqueName: \"kubernetes.io/projected/b92ad256-0ca7-4282-9991-a67bcc91ca18-kube-api-access-mn6cq\") pod \"maas-api-revoke-sub-5069c276-tjl44\" (UID: \"b92ad256-0ca7-4282-9991-a67bcc91ca18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:48.145308 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.145284    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88591333-4k9bc\" (UID: \"d0729dd4-778e-4dcd-af53-be08c31754fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:48.145881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.145852    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88591333-4k9bc\" (UID: \"d0729dd4-778e-4dcd-af53-be08c31754fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:48.147348 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.147319    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88591333-4k9bc\" (UID: \"d0729dd4-778e-4dcd-af53-be08c31754fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:48.168428 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.168406    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzs5b\" (UniqueName: \"kubernetes.io/projected/d0729dd4-778e-4dcd-af53-be08c31754fa-kube-api-access-gzs5b\") pod \"maas-api-revoke-sub-88591333-4k9bc\" (UID: \"d0729dd4-778e-4dcd-af53-be08c31754fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:48.216454 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.216416    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:48.246570 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.246534    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5069c276-tjl44\" (UID: \"b92ad256-0ca7-4282-9991-a67bcc91ca18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:48.246717 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.246604    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn6cq\" (UniqueName: \"kubernetes.io/projected/b92ad256-0ca7-4282-9991-a67bcc91ca18-kube-api-access-mn6cq\") pod \"maas-api-revoke-sub-5069c276-tjl44\" (UID: \"b92ad256-0ca7-4282-9991-a67bcc91ca18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:48.246775 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.246718    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5069c276-tjl44\" (UID: \"b92ad256-0ca7-4282-9991-a67bcc91ca18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:48.247365 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.247337    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5069c276-tjl44\" (UID: \"b92ad256-0ca7-4282-9991-a67bcc91ca18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:48.249009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.248988    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5069c276-tjl44\" (UID: \"b92ad256-0ca7-4282-9991-a67bcc91ca18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:48.272048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.270756    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn6cq\" (UniqueName: \"kubernetes.io/projected/b92ad256-0ca7-4282-9991-a67bcc91ca18-kube-api-access-mn6cq\") pod \"maas-api-revoke-sub-5069c276-tjl44\" (UID: \"b92ad256-0ca7-4282-9991-a67bcc91ca18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:48.365237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.365197    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:48.537535 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:50:48.537505    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0729dd4_778e_4dcd_af53_be08c31754fa.slice/crio-af007da3a6e1b599713877c0b84b7803a40ad0bfe9d7c7e617933f64baec0f42 WatchSource:0}: Error finding container af007da3a6e1b599713877c0b84b7803a40ad0bfe9d7c7e617933f64baec0f42: Status 404 returned error can't find the container with id af007da3a6e1b599713877c0b84b7803a40ad0bfe9d7c7e617933f64baec0f42
Nov 02 07:50:48.540705 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.540678    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"]
Nov 02 07:50:48.676987 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:50:48.676949    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92ad256_0ca7_4282_9991_a67bcc91ca18.slice/crio-2a74a1a63d39ccfddb6bd5014c58924169b7191341e60d004250cfad6e114c4d WatchSource:0}: Error finding container 2a74a1a63d39ccfddb6bd5014c58924169b7191341e60d004250cfad6e114c4d: Status 404 returned error can't find the container with id 2a74a1a63d39ccfddb6bd5014c58924169b7191341e60d004250cfad6e114c4d
Nov 02 07:50:48.678553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:48.678524    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"]
Nov 02 07:50:49.025856 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:49.025804    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc" event={"ID":"d0729dd4-778e-4dcd-af53-be08c31754fa","Type":"ContainerStarted","Data":"75a2c373aabac59de356286ef11de0eda864443936490f6b98c8a0503bdc087a"}
Nov 02 07:50:49.026196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:49.026155    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc" event={"ID":"d0729dd4-778e-4dcd-af53-be08c31754fa","Type":"ContainerStarted","Data":"af007da3a6e1b599713877c0b84b7803a40ad0bfe9d7c7e617933f64baec0f42"}
Nov 02 07:50:49.027672 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:49.027647    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44" event={"ID":"b92ad256-0ca7-4282-9991-a67bcc91ca18","Type":"ContainerStarted","Data":"2a74a1a63d39ccfddb6bd5014c58924169b7191341e60d004250cfad6e114c4d"}
Nov 02 07:50:49.029897 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:49.029857    4898 generic.go:356] "Generic (PLEG): container finished" podID="42283595-1739-4f03-845c-41f9ef5f5073" containerID="bf78c593adae2369c199bdc0b6a968ed6128774808343cc3a379bd382e34c17a" exitCode=0
Nov 02 07:50:49.029990 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:49.029918    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" event={"ID":"42283595-1739-4f03-845c-41f9ef5f5073","Type":"ContainerDied","Data":"bf78c593adae2369c199bdc0b6a968ed6128774808343cc3a379bd382e34c17a"}
Nov 02 07:50:49.073626 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:49.073578    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc" podStartSLOduration=2.073561974 podStartE2EDuration="2.073561974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:50:49.070311251 +0000 UTC m=+1599.760638216" watchObservedRunningTime="2026-11-02 07:50:49.073561974 +0000 UTC m=+1599.763888935"
Nov 02 07:50:49.915025 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:49.909133    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:50:49.974906 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:49.974106    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-data\") pod \"42283595-1739-4f03-845c-41f9ef5f5073\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") "
Nov 02 07:50:49.974906 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:49.974242    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42283595-1739-4f03-845c-41f9ef5f5073-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42283595-1739-4f03-845c-41f9ef5f5073-istio-podinfo\") pod \"42283595-1739-4f03-845c-41f9ef5f5073\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") "
Nov 02 07:50:49.974906 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:49.974332    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-kube-api-access-jrm26\" (UniqueName: \"kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-kube-api-access-jrm26\") pod \"42283595-1739-4f03-845c-41f9ef5f5073\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") "
Nov 02 07:50:49.974906 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:49.974742    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-data" pod "42283595-1739-4f03-845c-41f9ef5f5073" (UID: "42283595-1739-4f03-845c-41f9ef5f5073"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:49.975251 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:49.975007    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:50.080914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.080424    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-credential-socket\") pod \"42283595-1739-4f03-845c-41f9ef5f5073\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") "
Nov 02 07:50:50.080914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.080534    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-certs\") pod \"42283595-1739-4f03-845c-41f9ef5f5073\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") "
Nov 02 07:50:50.080914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.080655    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-istio-token\" (UniqueName: \"kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-istio-token\") pod \"42283595-1739-4f03-845c-41f9ef5f5073\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") "
Nov 02 07:50:50.080914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.080696    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-envoy\") pod \"42283595-1739-4f03-845c-41f9ef5f5073\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") "
Nov 02 07:50:50.081300 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.081161    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-socket\") pod \"42283595-1739-4f03-845c-41f9ef5f5073\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") "
Nov 02 07:50:50.081300 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.081227    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42283595-1739-4f03-845c-41f9ef5f5073-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42283595-1739-4f03-845c-41f9ef5f5073-istiod-ca-cert\") pod \"42283595-1739-4f03-845c-41f9ef5f5073\" (UID: \"42283595-1739-4f03-845c-41f9ef5f5073\") "
Nov 02 07:50:50.085456 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.082505    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42283595-1739-4f03-845c-41f9ef5f5073-istiod-ca-cert" pod "42283595-1739-4f03-845c-41f9ef5f5073" (UID: "42283595-1739-4f03-845c-41f9ef5f5073"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:50.085456 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.082908    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-credential-socket" pod "42283595-1739-4f03-845c-41f9ef5f5073" (UID: "42283595-1739-4f03-845c-41f9ef5f5073"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:50.085456 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.083121    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-certs" pod "42283595-1739-4f03-845c-41f9ef5f5073" (UID: "42283595-1739-4f03-845c-41f9ef5f5073"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:50.090316 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.086235    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42283595-1739-4f03-845c-41f9ef5f5073-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:50.090316 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.086261    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:50.090316 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.086276    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:50.091004 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.090971    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-socket" pod "42283595-1739-4f03-845c-41f9ef5f5073" (UID: "42283595-1739-4f03-845c-41f9ef5f5073"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:50.092207 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.092157    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-istio-token" pod "42283595-1739-4f03-845c-41f9ef5f5073" (UID: "42283595-1739-4f03-845c-41f9ef5f5073"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:50.097165 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.097134    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-envoy" pod "42283595-1739-4f03-845c-41f9ef5f5073" (UID: "42283595-1739-4f03-845c-41f9ef5f5073"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:50.108912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.108864    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42283595-1739-4f03-845c-41f9ef5f5073-istio-podinfo" pod "42283595-1739-4f03-845c-41f9ef5f5073" (UID: "42283595-1739-4f03-845c-41f9ef5f5073"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:50:50.142914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.131483    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" event={"ID":"42283595-1739-4f03-845c-41f9ef5f5073","Type":"ContainerDied","Data":"8bf25ed73fb8b37214e203277baa4f2348e2e53a3c650266d6f5a0b1b4a5e764"}
Nov 02 07:50:50.142914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.131557    4898 scope.go:122] "RemoveContainer" containerID="bf78c593adae2369c199bdc0b6a968ed6128774808343cc3a379bd382e34c17a"
Nov 02 07:50:50.142914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.131719    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"
Nov 02 07:50:50.142914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.135867    4898 generic.go:356] "Generic (PLEG): container finished" podID="d0729dd4-778e-4dcd-af53-be08c31754fa" containerID="75a2c373aabac59de356286ef11de0eda864443936490f6b98c8a0503bdc087a" exitCode=0
Nov 02 07:50:50.142914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.136113    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc" event={"ID":"d0729dd4-778e-4dcd-af53-be08c31754fa","Type":"ContainerDied","Data":"75a2c373aabac59de356286ef11de0eda864443936490f6b98c8a0503bdc087a"}
Nov 02 07:50:50.142914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.138499    4898 generic.go:356] "Generic (PLEG): container finished" podID="b92ad256-0ca7-4282-9991-a67bcc91ca18" containerID="d468986da821c245b657ea0b67ee7993663e616add4691625c7d3ab8b4387545" exitCode=0
Nov 02 07:50:50.142914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.138623    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44" event={"ID":"b92ad256-0ca7-4282-9991-a67bcc91ca18","Type":"ContainerDied","Data":"d468986da821c245b657ea0b67ee7993663e616add4691625c7d3ab8b4387545"}
Nov 02 07:50:50.142914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.139035    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-kube-api-access-jrm26" pod "42283595-1739-4f03-845c-41f9ef5f5073" (UID: "42283595-1739-4f03-845c-41f9ef5f5073"). InnerVolumeSpecName "kube-api-access-jrm26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:50.192344 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.191871    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42283595-1739-4f03-845c-41f9ef5f5073-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:50.192344 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.191923    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrm26\" (UniqueName: \"kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-kube-api-access-jrm26\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:50.192344 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.191941    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:50.192344 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.191953    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42283595-1739-4f03-845c-41f9ef5f5073-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:50.192344 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.191964    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42283595-1739-4f03-845c-41f9ef5f5073-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:50.568917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.566942    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"]
Nov 02 07:50:50.598399 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.598365    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm"]
Nov 02 07:50:50.836197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.835727    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:50:50.836197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:50.835790    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-nbsgm" podUID="42283595-1739-4f03-845c-41f9ef5f5073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:50:51.115957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:51.115395    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"]
Nov 02 07:50:51.115957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:51.115654    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" podUID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" containerName="istio-proxy" containerID="cri-o://6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965" gracePeriod=30
Nov 02 07:50:51.705966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:51.681000    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42283595-1739-4f03-845c-41f9ef5f5073" path="/var/lib/kubelet/pods/42283595-1739-4f03-845c-41f9ef5f5073/volumes"
Nov 02 07:50:51.892423 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:51.888638    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:50:51.948919 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:51.945852    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:50:51.948919 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:51.946116    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 02 07:50:52.032053 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:52.028986    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:50:52.977792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:52.952227    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:50:53.012849 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.012812    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:50:53.046458 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.046414    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc" event={"ID":"d0729dd4-778e-4dcd-af53-be08c31754fa","Type":"ContainerDied","Data":"af007da3a6e1b599713877c0b84b7803a40ad0bfe9d7c7e617933f64baec0f42"}
Nov 02 07:50:53.046645 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.046487    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af007da3a6e1b599713877c0b84b7803a40ad0bfe9d7c7e617933f64baec0f42"
Nov 02 07:50:53.046769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.046749    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:50:53.048433 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.048405    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:53.049270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.049243    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44" event={"ID":"b92ad256-0ca7-4282-9991-a67bcc91ca18","Type":"ContainerDied","Data":"2a74a1a63d39ccfddb6bd5014c58924169b7191341e60d004250cfad6e114c4d"}
Nov 02 07:50:53.049270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.049290    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a74a1a63d39ccfddb6bd5014c58924169b7191341e60d004250cfad6e114c4d"
Nov 02 07:50:53.050769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.050748    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:53.051699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.051670    4898 generic.go:356] "Generic (PLEG): container finished" podID="53de2d77-31bc-415c-8c29-d768438375af" containerID="812e1dab2a4d367460238b96357e1ec9af08e4f88bbb93298a73c4157bff655b" exitCode=0
Nov 02 07:50:53.051820 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.051800    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" event={"ID":"53de2d77-31bc-415c-8c29-d768438375af","Type":"ContainerDied","Data":"812e1dab2a4d367460238b96357e1ec9af08e4f88bbb93298a73c4157bff655b"}
Nov 02 07:50:53.052028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.052009    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt" event={"ID":"53de2d77-31bc-415c-8c29-d768438375af","Type":"ContainerDied","Data":"8359db1f22a3c88efd9cafd4fa3a7d8875aa26cf75bf390ca52438071d96c922"}
Nov 02 07:50:53.052161 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.052144    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8359db1f22a3c88efd9cafd4fa3a7d8875aa26cf75bf390ca52438071d96c922"
Nov 02 07:50:53.052286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.052274    4898 scope.go:122] "RemoveContainer" containerID="dc11653148133e1dbbce433fdadc2fca8b60c9be24672213d9897bc89c53322d"
Nov 02 07:50:53.053960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.053863    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:50:53.054796 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.054771    4898 generic.go:356] "Generic (PLEG): container finished" podID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerID="492134a347ade079655780fde0d851f7ce36f495a050eb533c2ec6930e3f6f6d" exitCode=0
Nov 02 07:50:53.055019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.054999    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" event={"ID":"0738e017-3fb4-4004-aa44-671d2a0a4400","Type":"ContainerDied","Data":"492134a347ade079655780fde0d851f7ce36f495a050eb533c2ec6930e3f6f6d"}
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.108550    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-data\") pod \"53de2d77-31bc-415c-8c29-d768438375af\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.108622    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0729dd4-778e-4dcd-af53-be08c31754fa-kube-api-access-gzs5b\" (UniqueName: \"kubernetes.io/projected/d0729dd4-778e-4dcd-af53-be08c31754fa-kube-api-access-gzs5b\") pod \"d0729dd4-778e-4dcd-af53-be08c31754fa\" (UID: \"d0729dd4-778e-4dcd-af53-be08c31754fa\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.108660    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-envoy\") pod \"53de2d77-31bc-415c-8c29-d768438375af\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.108714    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53de2d77-31bc-415c-8c29-d768438375af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53de2d77-31bc-415c-8c29-d768438375af-istiod-ca-cert\") pod \"53de2d77-31bc-415c-8c29-d768438375af\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.108774    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-openshift-ca\") pod \"b92ad256-0ca7-4282-9991-a67bcc91ca18\" (UID: \"b92ad256-0ca7-4282-9991-a67bcc91ca18\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.108811    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-istio-token\" (UniqueName: \"kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-istio-token\") pod \"53de2d77-31bc-415c-8c29-d768438375af\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.108850    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92ad256-0ca7-4282-9991-a67bcc91ca18-kube-api-access-mn6cq\" (UniqueName: \"kubernetes.io/projected/b92ad256-0ca7-4282-9991-a67bcc91ca18-kube-api-access-mn6cq\") pod \"b92ad256-0ca7-4282-9991-a67bcc91ca18\" (UID: \"b92ad256-0ca7-4282-9991-a67bcc91ca18\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.108957    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-socket\") pod \"53de2d77-31bc-415c-8c29-d768438375af\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.109004    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-certs\") pod \"53de2d77-31bc-415c-8c29-d768438375af\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.109038    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-openshift-ca\") pod \"d0729dd4-778e-4dcd-af53-be08c31754fa\" (UID: \"d0729dd4-778e-4dcd-af53-be08c31754fa\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.109121    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53de2d77-31bc-415c-8c29-d768438375af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53de2d77-31bc-415c-8c29-d768438375af-istio-podinfo\") pod \"53de2d77-31bc-415c-8c29-d768438375af\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.109158    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-xks-ca\") pod \"b92ad256-0ca7-4282-9991-a67bcc91ca18\" (UID: \"b92ad256-0ca7-4282-9991-a67bcc91ca18\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.109216    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-kube-api-access-qkqbk\" (UniqueName: \"kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-kube-api-access-qkqbk\") pod \"53de2d77-31bc-415c-8c29-d768438375af\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.109260    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-credential-socket\") pod \"53de2d77-31bc-415c-8c29-d768438375af\" (UID: \"53de2d77-31bc-415c-8c29-d768438375af\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.109322    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-xks-ca\") pod \"d0729dd4-778e-4dcd-af53-be08c31754fa\" (UID: \"d0729dd4-778e-4dcd-af53-be08c31754fa\") "
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.110587    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-certs" pod "53de2d77-31bc-415c-8c29-d768438375af" (UID: "53de2d77-31bc-415c-8c29-d768438375af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.110975    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-data" pod "53de2d77-31bc-415c-8c29-d768438375af" (UID: "53de2d77-31bc-415c-8c29-d768438375af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:53.114907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.111331    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-openshift-ca" pod "d0729dd4-778e-4dcd-af53-be08c31754fa" (UID: "d0729dd4-778e-4dcd-af53-be08c31754fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:53.117377 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.117353    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-credential-socket" pod "53de2d77-31bc-415c-8c29-d768438375af" (UID: "53de2d77-31bc-415c-8c29-d768438375af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:53.118511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.118489    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-openshift-ca" pod "b92ad256-0ca7-4282-9991-a67bcc91ca18" (UID: "b92ad256-0ca7-4282-9991-a67bcc91ca18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:53.125676 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.125655    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53de2d77-31bc-415c-8c29-d768438375af-istiod-ca-cert" pod "53de2d77-31bc-415c-8c29-d768438375af" (UID: "53de2d77-31bc-415c-8c29-d768438375af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:53.127038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.125850    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-envoy" pod "53de2d77-31bc-415c-8c29-d768438375af" (UID: "53de2d77-31bc-415c-8c29-d768438375af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:53.130453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.126493    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-istio-token" pod "53de2d77-31bc-415c-8c29-d768438375af" (UID: "53de2d77-31bc-415c-8c29-d768438375af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:53.130565 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.128609    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-kube-api-access-qkqbk" pod "53de2d77-31bc-415c-8c29-d768438375af" (UID: "53de2d77-31bc-415c-8c29-d768438375af"). InnerVolumeSpecName "kube-api-access-qkqbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:53.130642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.128792    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-socket" pod "53de2d77-31bc-415c-8c29-d768438375af" (UID: "53de2d77-31bc-415c-8c29-d768438375af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:53.135381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.135356    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53de2d77-31bc-415c-8c29-d768438375af-istio-podinfo" pod "53de2d77-31bc-415c-8c29-d768438375af" (UID: "53de2d77-31bc-415c-8c29-d768438375af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:50:53.138430 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.138392    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92ad256-0ca7-4282-9991-a67bcc91ca18-kube-api-access-mn6cq" pod "b92ad256-0ca7-4282-9991-a67bcc91ca18" (UID: "b92ad256-0ca7-4282-9991-a67bcc91ca18"). InnerVolumeSpecName "kube-api-access-mn6cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:53.140422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.140396    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0729dd4-778e-4dcd-af53-be08c31754fa-kube-api-access-gzs5b" pod "d0729dd4-778e-4dcd-af53-be08c31754fa" (UID: "d0729dd4-778e-4dcd-af53-be08c31754fa"). InnerVolumeSpecName "kube-api-access-gzs5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:53.140784 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.140750    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-xks-ca" pod "d0729dd4-778e-4dcd-af53-be08c31754fa" (UID: "d0729dd4-778e-4dcd-af53-be08c31754fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:53.141009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.140977    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-xks-ca" pod "b92ad256-0ca7-4282-9991-a67bcc91ca18" (UID: "b92ad256-0ca7-4282-9991-a67bcc91ca18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212436    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn6cq\" (UniqueName: \"kubernetes.io/projected/b92ad256-0ca7-4282-9991-a67bcc91ca18-kube-api-access-mn6cq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212471    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212537    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212555    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212571    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53de2d77-31bc-415c-8c29-d768438375af-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212585    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212598    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkqbk\" (UniqueName: \"kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-kube-api-access-qkqbk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212611    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212625    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0729dd4-778e-4dcd-af53-be08c31754fa-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212638    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212650    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzs5b\" (UniqueName: \"kubernetes.io/projected/d0729dd4-778e-4dcd-af53-be08c31754fa-kube-api-access-gzs5b\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212664    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53de2d77-31bc-415c-8c29-d768438375af-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212677    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53de2d77-31bc-415c-8c29-d768438375af-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212689    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92ad256-0ca7-4282-9991-a67bcc91ca18-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.212825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.212702    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53de2d77-31bc-415c-8c29-d768438375af-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.256326 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.256295    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:50:53.416863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.416758    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-data\") pod \"0738e017-3fb4-4004-aa44-671d2a0a4400\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") "
Nov 02 07:50:53.416863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.416823    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-token\" (UniqueName: \"kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-token\") pod \"0738e017-3fb4-4004-aa44-671d2a0a4400\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") "
Nov 02 07:50:53.417130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.417006    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0738e017-3fb4-4004-aa44-671d2a0a4400-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0738e017-3fb4-4004-aa44-671d2a0a4400-istiod-ca-cert\") pod \"0738e017-3fb4-4004-aa44-671d2a0a4400\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") "
Nov 02 07:50:53.417130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.417072    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-podinfo\") pod \"0738e017-3fb4-4004-aa44-671d2a0a4400\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") "
Nov 02 07:50:53.417130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.417104    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-certs\") pod \"0738e017-3fb4-4004-aa44-671d2a0a4400\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") "
Nov 02 07:50:53.417300 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.417153    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-kube-api-access-2zw5f\" (UniqueName: \"kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-kube-api-access-2zw5f\") pod \"0738e017-3fb4-4004-aa44-671d2a0a4400\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") "
Nov 02 07:50:53.417300 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.417210    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-credential-socket\") pod \"0738e017-3fb4-4004-aa44-671d2a0a4400\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") "
Nov 02 07:50:53.417300 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.417239    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-socket\") pod \"0738e017-3fb4-4004-aa44-671d2a0a4400\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") "
Nov 02 07:50:53.417584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.417539    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-certs" pod "0738e017-3fb4-4004-aa44-671d2a0a4400" (UID: "0738e017-3fb4-4004-aa44-671d2a0a4400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:53.418105 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.417871    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-socket" pod "0738e017-3fb4-4004-aa44-671d2a0a4400" (UID: "0738e017-3fb4-4004-aa44-671d2a0a4400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:53.418105 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.417944    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0738e017-3fb4-4004-aa44-671d2a0a4400-istiod-ca-cert" pod "0738e017-3fb4-4004-aa44-671d2a0a4400" (UID: "0738e017-3fb4-4004-aa44-671d2a0a4400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:53.418105 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.417974    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-data" pod "0738e017-3fb4-4004-aa44-671d2a0a4400" (UID: "0738e017-3fb4-4004-aa44-671d2a0a4400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:53.418105 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.418045    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-credential-socket" pod "0738e017-3fb4-4004-aa44-671d2a0a4400" (UID: "0738e017-3fb4-4004-aa44-671d2a0a4400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:53.419382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.418402    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-envoy\") pod \"0738e017-3fb4-4004-aa44-671d2a0a4400\" (UID: \"0738e017-3fb4-4004-aa44-671d2a0a4400\") "
Nov 02 07:50:53.419382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.419106    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.419382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.419127    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0738e017-3fb4-4004-aa44-671d2a0a4400-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.419382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.419139    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.419382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.419152    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.419382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.419166    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.420723 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.420697    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-envoy" pod "0738e017-3fb4-4004-aa44-671d2a0a4400" (UID: "0738e017-3fb4-4004-aa44-671d2a0a4400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:53.420808 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.420725    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-podinfo" pod "0738e017-3fb4-4004-aa44-671d2a0a4400" (UID: "0738e017-3fb4-4004-aa44-671d2a0a4400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:50:53.421558 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.421238    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-kube-api-access-2zw5f" pod "0738e017-3fb4-4004-aa44-671d2a0a4400" (UID: "0738e017-3fb4-4004-aa44-671d2a0a4400"). InnerVolumeSpecName "kube-api-access-2zw5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:53.421783 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.421753    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-token" pod "0738e017-3fb4-4004-aa44-671d2a0a4400" (UID: "0738e017-3fb4-4004-aa44-671d2a0a4400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:53.520152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.520066    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.520152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.520099    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zw5f\" (UniqueName: \"kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-kube-api-access-2zw5f\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.520152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.520110    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.520152 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.520118    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0738e017-3fb4-4004-aa44-671d2a0a4400-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:53.986964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:53.986930    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"
Nov 02 07:50:54.068720 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:54.068683    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"
Nov 02 07:50:54.070581 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:54.070546    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf" event={"ID":"0738e017-3fb4-4004-aa44-671d2a0a4400","Type":"ContainerDied","Data":"584011f90dfedca24c5d8982b7a6d877308c5a662fbd3520ee4c4e00823b8007"}
Nov 02 07:50:54.070581 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:54.070578    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"
Nov 02 07:50:54.070827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:54.070607    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"
Nov 02 07:50:54.070827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:54.070620    4898 scope.go:122] "RemoveContainer" containerID="492134a347ade079655780fde0d851f7ce36f495a050eb533c2ec6930e3f6f6d"
Nov 02 07:50:54.071082 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:54.071065    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"
Nov 02 07:50:54.284953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:54.282948    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"]
Nov 02 07:50:54.292429 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:54.292379    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-f89vf"]
Nov 02 07:50:54.303467 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:54.303435    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"]
Nov 02 07:50:54.312126 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:54.312068    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-zhkkt"]
Nov 02 07:50:55.465520 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:55.465486    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" path="/var/lib/kubelet/pods/0738e017-3fb4-4004-aa44-671d2a0a4400/volumes"
Nov 02 07:50:55.466029 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:55.466014    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53de2d77-31bc-415c-8c29-d768438375af" path="/var/lib/kubelet/pods/53de2d77-31bc-415c-8c29-d768438375af/volumes"
Nov 02 07:50:56.465299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.465272    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:50:56.513865 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.513751    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-credential-socket\") pod \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") "
Nov 02 07:50:56.513865 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.513855    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-kube-api-access-fvf8b\" (UniqueName: \"kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-kube-api-access-fvf8b\") pod \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") "
Nov 02 07:50:56.514449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.513936    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-envoy\") pod \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") "
Nov 02 07:50:56.514449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514030    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-credential-socket" pod "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" (UID: "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:56.514449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514081    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-certs\") pod \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") "
Nov 02 07:50:56.514449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514119    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-socket\") pod \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") "
Nov 02 07:50:56.514449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514163    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-token\" (UniqueName: \"kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-token\") pod \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") "
Nov 02 07:50:56.514449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514204    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istiod-ca-cert\") pod \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") "
Nov 02 07:50:56.514449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514261    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-podinfo\") pod \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") "
Nov 02 07:50:56.514449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514295    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-data\") pod \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\" (UID: \"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137\") "
Nov 02 07:50:56.514449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514368    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-certs" pod "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" (UID: "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:56.515116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514664    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-data" pod "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" (UID: "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:56.515116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514868    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.515116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514918    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.515116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514923    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istiod-ca-cert" pod "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" (UID: "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:50:56.515116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.514933    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.515116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.515111    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-socket" pod "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" (UID: "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:56.516501 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.516481    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-kube-api-access-fvf8b" pod "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" (UID: "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137"). InnerVolumeSpecName "kube-api-access-fvf8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:56.516642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.516622    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-envoy" pod "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" (UID: "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:50:56.516712 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.516654    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-podinfo" pod "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" (UID: "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:50:56.517003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.516976    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-token" pod "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" (UID: "4a340f3b-b2c0-429d-b55a-4d0b5b6b4137"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:50:56.615968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.615935    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.615968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.615967    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.615968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.615976    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.616228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.615984    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.616228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.615993    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvf8b\" (UniqueName: \"kubernetes.io/projected/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-kube-api-access-fvf8b\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:56.616228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:56.616002    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:50:57.098130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:57.098098    4898 generic.go:356] "Generic (PLEG): container finished" podID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" containerID="6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965" exitCode=0
Nov 02 07:50:57.098333 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:57.098160    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"
Nov 02 07:50:57.098333 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:57.098178    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" event={"ID":"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137","Type":"ContainerDied","Data":"6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965"}
Nov 02 07:50:57.098333 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:57.098231    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl" event={"ID":"4a340f3b-b2c0-429d-b55a-4d0b5b6b4137","Type":"ContainerDied","Data":"76293efae74c1e02fc240d86ebbc150a0e5f55a66b3eebee7adf078bb2514c7c"}
Nov 02 07:50:57.098333 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:57.098255    4898 scope.go:122] "RemoveContainer" containerID="6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965"
Nov 02 07:50:57.113744 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:57.113727    4898 scope.go:122] "RemoveContainer" containerID="6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965"
Nov 02 07:50:57.114108 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:50:57.114091    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965\": container with ID starting with 6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965 not found: ID does not exist" containerID="6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965"
Nov 02 07:50:57.114231 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:57.114120    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965"} err="failed to get container status \"6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965\": rpc error: code = NotFound desc = could not find container \"6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965\": container with ID starting with 6bd7a0c947ba25b2ae2daa954ced9ed14c90a2247beef745c29dbac7781c4965 not found: ID does not exist"
Nov 02 07:50:57.164575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:57.164545    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"]
Nov 02 07:50:57.177316 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:57.177281    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-626zl"]
Nov 02 07:50:57.466510 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:57.466473    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" path="/var/lib/kubelet/pods/4a340f3b-b2c0-429d-b55a-4d0b5b6b4137/volumes"
Nov 02 07:50:58.315544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.315511    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"]
Nov 02 07:50:58.316220 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316200    4898 cpu_manager.go:414] "containerMap: removing container" podUID="53de2d77-31bc-415c-8c29-d768438375af" containerName="istio-proxy"
Nov 02 07:50:58.316220 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316218    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53de2d77-31bc-415c-8c29-d768438375af" containerName="istio-proxy"
Nov 02 07:50:58.316368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316237    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d0729dd4-778e-4dcd-af53-be08c31754fa" containerName="revoke-keys"
Nov 02 07:50:58.316368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316242    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0729dd4-778e-4dcd-af53-be08c31754fa" containerName="revoke-keys"
Nov 02 07:50:58.316368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316249    4898 cpu_manager.go:414] "containerMap: removing container" podUID="53de2d77-31bc-415c-8c29-d768438375af" containerName="istio-proxy"
Nov 02 07:50:58.316368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316254    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53de2d77-31bc-415c-8c29-d768438375af" containerName="istio-proxy"
Nov 02 07:50:58.316368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316268    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b92ad256-0ca7-4282-9991-a67bcc91ca18" containerName="revoke-keys"
Nov 02 07:50:58.316368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316273    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92ad256-0ca7-4282-9991-a67bcc91ca18" containerName="revoke-keys"
Nov 02 07:50:58.316368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316294    4898 cpu_manager.go:414] "containerMap: removing container" podUID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" containerName="istio-proxy"
Nov 02 07:50:58.316368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316299    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" containerName="istio-proxy"
Nov 02 07:50:58.316368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316312    4898 cpu_manager.go:414] "containerMap: removing container" podUID="42283595-1739-4f03-845c-41f9ef5f5073" containerName="istio-proxy"
Nov 02 07:50:58.316368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316317    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42283595-1739-4f03-845c-41f9ef5f5073" containerName="istio-proxy"
Nov 02 07:50:58.316368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316328    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerName="istio-proxy"
Nov 02 07:50:58.316368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316332    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerName="istio-proxy"
Nov 02 07:50:58.316749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316433    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="42283595-1739-4f03-845c-41f9ef5f5073" containerName="istio-proxy"
Nov 02 07:50:58.316749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316444    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0738e017-3fb4-4004-aa44-671d2a0a4400" containerName="istio-proxy"
Nov 02 07:50:58.316749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316452    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="53de2d77-31bc-415c-8c29-d768438375af" containerName="istio-proxy"
Nov 02 07:50:58.316749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316458    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0729dd4-778e-4dcd-af53-be08c31754fa" containerName="revoke-keys"
Nov 02 07:50:58.316749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316465    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="53de2d77-31bc-415c-8c29-d768438375af" containerName="istio-proxy"
Nov 02 07:50:58.316749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316472    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92ad256-0ca7-4282-9991-a67bcc91ca18" containerName="revoke-keys"
Nov 02 07:50:58.316749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.316482    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a340f3b-b2c0-429d-b55a-4d0b5b6b4137" containerName="istio-proxy"
Nov 02 07:50:58.321186 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.321166    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:50:58.350097 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.350066    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:50:58.352224 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.352138    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"]
Nov 02 07:50:58.354256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.354236    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:50:58.354800 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.354778    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:50:58.440269 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.440241    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:50:58.443073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.443042    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb7e782b-wk475\" (UID: \"7d25f037-09e4-49c4-a749-0e9715806f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:50:58.443257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.443231    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb7e782b-wk475\" (UID: \"7d25f037-09e4-49c4-a749-0e9715806f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:50:58.443310 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.443294    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2vgg\" (UniqueName: \"kubernetes.io/projected/7d25f037-09e4-49c4-a749-0e9715806f73-kube-api-access-r2vgg\") pod \"maas-api-revoke-sub-eb7e782b-wk475\" (UID: \"7d25f037-09e4-49c4-a749-0e9715806f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:50:58.544859 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.544828    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb7e782b-wk475\" (UID: \"7d25f037-09e4-49c4-a749-0e9715806f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:50:58.545046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.545024    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb7e782b-wk475\" (UID: \"7d25f037-09e4-49c4-a749-0e9715806f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:50:58.545112 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.545097    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2vgg\" (UniqueName: \"kubernetes.io/projected/7d25f037-09e4-49c4-a749-0e9715806f73-kube-api-access-r2vgg\") pod \"maas-api-revoke-sub-eb7e782b-wk475\" (UID: \"7d25f037-09e4-49c4-a749-0e9715806f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:50:58.545735 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.545712    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb7e782b-wk475\" (UID: \"7d25f037-09e4-49c4-a749-0e9715806f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:50:58.547256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.547238    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb7e782b-wk475\" (UID: \"7d25f037-09e4-49c4-a749-0e9715806f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:50:58.568696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.568635    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2vgg\" (UniqueName: \"kubernetes.io/projected/7d25f037-09e4-49c4-a749-0e9715806f73-kube-api-access-r2vgg\") pod \"maas-api-revoke-sub-eb7e782b-wk475\" (UID: \"7d25f037-09e4-49c4-a749-0e9715806f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:50:58.670090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.670054    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:50:58.977787 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:50:58.977756    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d25f037_09e4_49c4_a749_0e9715806f73.slice/crio-e573651d27950fddc8cb1163819ea5fda2d68ef6187a9b807cb149514bafa84b WatchSource:0}: Error finding container e573651d27950fddc8cb1163819ea5fda2d68ef6187a9b807cb149514bafa84b: Status 404 returned error can't find the container with id e573651d27950fddc8cb1163819ea5fda2d68ef6187a9b807cb149514bafa84b
Nov 02 07:50:58.982698 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:58.982670    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"]
Nov 02 07:50:59.122099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:50:59.122060    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475" event={"ID":"7d25f037-09e4-49c4-a749-0e9715806f73","Type":"ContainerStarted","Data":"e573651d27950fddc8cb1163819ea5fda2d68ef6187a9b807cb149514bafa84b"}
Nov 02 07:51:00.132917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:00.132708    4898 generic.go:356] "Generic (PLEG): container finished" podID="7d25f037-09e4-49c4-a749-0e9715806f73" containerID="d4f9001281526a2977276904a74e28de12998efcfb6f5f2332d729d2a3b01235" exitCode=0
Nov 02 07:51:00.133327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:00.132969    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475" event={"ID":"7d25f037-09e4-49c4-a749-0e9715806f73","Type":"ContainerDied","Data":"d4f9001281526a2977276904a74e28de12998efcfb6f5f2332d729d2a3b01235"}
Nov 02 07:51:00.946802 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:00.946757    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"]
Nov 02 07:51:00.947054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:00.947026    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy" containerID="cri-o://4d083dbf6877c556b912ca32ecff68020f4ed489889ed3dadc5a6e8caf95a5f3" gracePeriod=30
Nov 02 07:51:01.357911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:01.357867    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:51:01.492056 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:01.491975    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d25f037-09e4-49c4-a749-0e9715806f73-kube-api-access-r2vgg\" (UniqueName: \"kubernetes.io/projected/7d25f037-09e4-49c4-a749-0e9715806f73-kube-api-access-r2vgg\") pod \"7d25f037-09e4-49c4-a749-0e9715806f73\" (UID: \"7d25f037-09e4-49c4-a749-0e9715806f73\") "
Nov 02 07:51:01.492249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:01.492112    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-openshift-ca\") pod \"7d25f037-09e4-49c4-a749-0e9715806f73\" (UID: \"7d25f037-09e4-49c4-a749-0e9715806f73\") "
Nov 02 07:51:01.492320 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:01.492269    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-xks-ca\") pod \"7d25f037-09e4-49c4-a749-0e9715806f73\" (UID: \"7d25f037-09e4-49c4-a749-0e9715806f73\") "
Nov 02 07:51:01.492712 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:01.492682    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-openshift-ca" pod "7d25f037-09e4-49c4-a749-0e9715806f73" (UID: "7d25f037-09e4-49c4-a749-0e9715806f73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:51:01.494727 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:01.494696    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d25f037-09e4-49c4-a749-0e9715806f73-kube-api-access-r2vgg" pod "7d25f037-09e4-49c4-a749-0e9715806f73" (UID: "7d25f037-09e4-49c4-a749-0e9715806f73"). InnerVolumeSpecName "kube-api-access-r2vgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:51:01.494921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:01.494865    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-xks-ca" pod "7d25f037-09e4-49c4-a749-0e9715806f73" (UID: "7d25f037-09e4-49c4-a749-0e9715806f73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:51:01.593734 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:01.593695    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:01.593734 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:01.593725    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2vgg\" (UniqueName: \"kubernetes.io/projected/7d25f037-09e4-49c4-a749-0e9715806f73-kube-api-access-r2vgg\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:01.593734 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:01.593735    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d25f037-09e4-49c4-a749-0e9715806f73-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:02.164816 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:02.164782    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"
Nov 02 07:51:02.165049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:02.164806    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475" event={"ID":"7d25f037-09e4-49c4-a749-0e9715806f73","Type":"ContainerDied","Data":"e573651d27950fddc8cb1163819ea5fda2d68ef6187a9b807cb149514bafa84b"}
Nov 02 07:51:02.165049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:02.164923    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e573651d27950fddc8cb1163819ea5fda2d68ef6187a9b807cb149514bafa84b"
Nov 02 07:51:02.981688 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:02.981652    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"]
Nov 02 07:51:02.982669 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:02.982644    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7d25f037-09e4-49c4-a749-0e9715806f73" containerName="revoke-keys"
Nov 02 07:51:02.982726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:02.982674    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d25f037-09e4-49c4-a749-0e9715806f73" containerName="revoke-keys"
Nov 02 07:51:02.982858 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:02.982844    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d25f037-09e4-49c4-a749-0e9715806f73" containerName="revoke-keys"
Nov 02 07:51:02.987353 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:02.987331    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.007135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.007103    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3624d8-openshift-default-dockercfg-ckcjf\""
Nov 02 07:51:03.040573 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.040543    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"]
Nov 02 07:51:03.115490 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.115456    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgkfz\" (UniqueName: \"kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-kube-api-access-wgkfz\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.115690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.115508    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-token\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.115690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.115531    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-envoy\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.115690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.115662    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-certs\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.115872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.115698    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-socket\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.115957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.115930    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-podinfo\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.116007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.115987    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-credential-socket\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.116072 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.116057    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76acab59-e07d-49ae-9075-0accdbe6e5b1-istiod-ca-cert\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.116115 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.116096    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-data\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.216861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.216829    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-certs\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.216869    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-socket\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.217015    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-podinfo\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.217042    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-credential-socket\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.217080    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76acab59-e07d-49ae-9075-0accdbe6e5b1-istiod-ca-cert\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.217110    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-data\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.217183    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgkfz\" (UniqueName: \"kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-kube-api-access-wgkfz\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.217238    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-token\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.217264    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-envoy\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217486 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.217319    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-certs\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217486 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.217359    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-socket\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217745 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.217718    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-credential-socket\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217842 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.217802    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76acab59-e07d-49ae-9075-0accdbe6e5b1-istiod-ca-cert\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.217842 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.217827    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-data\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.219578 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.219555    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-envoy\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.219679 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.219661    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-podinfo\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.242073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.242010    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgkfz\" (UniqueName: \"kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-kube-api-access-wgkfz\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.243236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.243219    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-token\") pod \"e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") " pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.309390 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.309358    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:03.729912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.717223    4898 patch_prober.go:29] interesting pod/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:51:03.729912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.717285    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:51:03.805964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.803775    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"]
Nov 02 07:51:03.829145 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.829100    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:03.829337 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.829318    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:03.829505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:03.829489    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:04.194268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:04.194040    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" event={"ID":"76acab59-e07d-49ae-9075-0accdbe6e5b1","Type":"ContainerStarted","Data":"663ae3b37b0b77c21928ec2af6d43b955fd884efb9f0c5cbef4654b460da4987"}
Nov 02 07:51:04.194268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:04.194112    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" event={"ID":"76acab59-e07d-49ae-9075-0accdbe6e5b1","Type":"ContainerStarted","Data":"d560d131b143ee93dfea944f1f2dafc464ced442db072d4fc8bce2befc5206a2"}
Nov 02 07:51:04.267913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:04.263919    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" podStartSLOduration=2.263883246 podStartE2EDuration="2.263883246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:04.263210871 +0000 UTC m=+1614.953537834" watchObservedRunningTime="2026-11-02 07:51:04.263883246 +0000 UTC m=+1614.954210211"
Nov 02 07:51:04.314913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:04.314577    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:04.314913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:04.314692    4898 patch_prober.go:29] interesting pod/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:51:04.314913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:04.314733    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 07:51:05.345914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:05.344315    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:05.362032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:05.347577    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:05.362032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:05.355346    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:05.615100 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:05.615004    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:05.876965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:05.875984    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:06.036957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.032040    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:06.115913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.115368    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Nov 02 07:51:06.198911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.191056    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"]
Nov 02 07:51:06.238929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.236197    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"]
Nov 02 07:51:06.238929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.236330    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:06.271948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.267303    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:51:06.271948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.267484    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:51:06.271948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.267536    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:51:06.271948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.268126    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:51:06.365207 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.363942    4898 generic.go:356] "Generic (PLEG): container finished" podID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerID="4d083dbf6877c556b912ca32ecff68020f4ed489889ed3dadc5a6e8caf95a5f3" exitCode=0
Nov 02 07:51:06.365207 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.365015    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" event={"ID":"4e1d63fb-e287-42c3-8496-237e4556e5f4","Type":"ContainerDied","Data":"4d083dbf6877c556b912ca32ecff68020f4ed489889ed3dadc5a6e8caf95a5f3"}
Nov 02 07:51:06.365207 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.365075    4898 scope.go:122] "RemoveContainer" containerID="3d45d340c2f0282df9866496edf469e78c3439d3370e6cf0c98af99e1cdca6c2"
Nov 02 07:51:06.413064 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.413028    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l9qh\" (UniqueName: \"kubernetes.io/projected/99f95feb-7b41-431b-bdf4-ed81c4a58119-kube-api-access-8l9qh\") pod \"maas-api-revoke-sub-3cc37371-rl2r5\" (UID: \"99f95feb-7b41-431b-bdf4-ed81c4a58119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:06.413356 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.413328    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cc37371-rl2r5\" (UID: \"99f95feb-7b41-431b-bdf4-ed81c4a58119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:06.413506 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.413489    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cc37371-rl2r5\" (UID: \"99f95feb-7b41-431b-bdf4-ed81c4a58119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:06.487712 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.487679    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:51:06.515095 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515064    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-kube-api-access-qh9x2\" (UniqueName: \"kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-kube-api-access-qh9x2\") pod \"4e1d63fb-e287-42c3-8496-237e4556e5f4\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") "
Nov 02 07:51:06.515221 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515139    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-token\" (UniqueName: \"kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-token\") pod \"4e1d63fb-e287-42c3-8496-237e4556e5f4\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") "
Nov 02 07:51:06.515221 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515191    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-socket\") pod \"4e1d63fb-e287-42c3-8496-237e4556e5f4\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") "
Nov 02 07:51:06.515358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515243    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-podinfo\") pod \"4e1d63fb-e287-42c3-8496-237e4556e5f4\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") "
Nov 02 07:51:06.515358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515309    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-credential-socket\") pod \"4e1d63fb-e287-42c3-8496-237e4556e5f4\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") "
Nov 02 07:51:06.515463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515362    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-envoy\") pod \"4e1d63fb-e287-42c3-8496-237e4556e5f4\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") "
Nov 02 07:51:06.515463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515413    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e1d63fb-e287-42c3-8496-237e4556e5f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1d63fb-e287-42c3-8496-237e4556e5f4-istiod-ca-cert\") pod \"4e1d63fb-e287-42c3-8496-237e4556e5f4\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") "
Nov 02 07:51:06.515566 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515460    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-data\") pod \"4e1d63fb-e287-42c3-8496-237e4556e5f4\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") "
Nov 02 07:51:06.515652 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515624    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-credential-socket" pod "4e1d63fb-e287-42c3-8496-237e4556e5f4" (UID: "4e1d63fb-e287-42c3-8496-237e4556e5f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:51:06.515713 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515652    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-certs\") pod \"4e1d63fb-e287-42c3-8496-237e4556e5f4\" (UID: \"4e1d63fb-e287-42c3-8496-237e4556e5f4\") "
Nov 02 07:51:06.515713 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515660    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-socket" pod "4e1d63fb-e287-42c3-8496-237e4556e5f4" (UID: "4e1d63fb-e287-42c3-8496-237e4556e5f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:51:06.515988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515919    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-data" pod "4e1d63fb-e287-42c3-8496-237e4556e5f4" (UID: "4e1d63fb-e287-42c3-8496-237e4556e5f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:51:06.515988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515950    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1d63fb-e287-42c3-8496-237e4556e5f4-istiod-ca-cert" pod "4e1d63fb-e287-42c3-8496-237e4556e5f4" (UID: "4e1d63fb-e287-42c3-8496-237e4556e5f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:51:06.515988 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.515962    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-certs" pod "4e1d63fb-e287-42c3-8496-237e4556e5f4" (UID: "4e1d63fb-e287-42c3-8496-237e4556e5f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:51:06.516255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.516066    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l9qh\" (UniqueName: \"kubernetes.io/projected/99f95feb-7b41-431b-bdf4-ed81c4a58119-kube-api-access-8l9qh\") pod \"maas-api-revoke-sub-3cc37371-rl2r5\" (UID: \"99f95feb-7b41-431b-bdf4-ed81c4a58119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:06.516255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.516138    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cc37371-rl2r5\" (UID: \"99f95feb-7b41-431b-bdf4-ed81c4a58119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:06.516255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.516195    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cc37371-rl2r5\" (UID: \"99f95feb-7b41-431b-bdf4-ed81c4a58119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:06.516503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.516456    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:06.516503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.516476    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1d63fb-e287-42c3-8496-237e4556e5f4-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:06.516503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.516492    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:06.516674 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.516506    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:06.516674 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.516521    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:06.517104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.517082    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cc37371-rl2r5\" (UID: \"99f95feb-7b41-431b-bdf4-ed81c4a58119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:06.517873 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.517851    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-token" pod "4e1d63fb-e287-42c3-8496-237e4556e5f4" (UID: "4e1d63fb-e287-42c3-8496-237e4556e5f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:51:06.517962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.517875    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-kube-api-access-qh9x2" pod "4e1d63fb-e287-42c3-8496-237e4556e5f4" (UID: "4e1d63fb-e287-42c3-8496-237e4556e5f4"). InnerVolumeSpecName "kube-api-access-qh9x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:51:06.518040 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.518017    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-envoy" pod "4e1d63fb-e287-42c3-8496-237e4556e5f4" (UID: "4e1d63fb-e287-42c3-8496-237e4556e5f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:51:06.518100 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.518085    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-podinfo" pod "4e1d63fb-e287-42c3-8496-237e4556e5f4" (UID: "4e1d63fb-e287-42c3-8496-237e4556e5f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:51:06.518775 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.518756    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cc37371-rl2r5\" (UID: \"99f95feb-7b41-431b-bdf4-ed81c4a58119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:06.541204 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.541180    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l9qh\" (UniqueName: \"kubernetes.io/projected/99f95feb-7b41-431b-bdf4-ed81c4a58119-kube-api-access-8l9qh\") pod \"maas-api-revoke-sub-3cc37371-rl2r5\" (UID: \"99f95feb-7b41-431b-bdf4-ed81c4a58119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:06.574507 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.574478    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:06.618000 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.617965    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh9x2\" (UniqueName: \"kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-kube-api-access-qh9x2\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:06.618126 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.618004    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:06.618126 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.618019    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:06.618126 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.618032    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1d63fb-e287-42c3-8496-237e4556e5f4-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:06.895299 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:51:06.895262    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f95feb_7b41_431b_bdf4_ed81c4a58119.slice/crio-aa7296dddeff41d627003df56315e99eeefcf9949f4b85f3e4bce0a00756a01b WatchSource:0}: Error finding container aa7296dddeff41d627003df56315e99eeefcf9949f4b85f3e4bce0a00756a01b: Status 404 returned error can't find the container with id aa7296dddeff41d627003df56315e99eeefcf9949f4b85f3e4bce0a00756a01b
Nov 02 07:51:06.897120 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:06.897095    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"]
Nov 02 07:51:07.375716 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:07.375678    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"
Nov 02 07:51:07.375716 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:07.375707    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z" event={"ID":"4e1d63fb-e287-42c3-8496-237e4556e5f4","Type":"ContainerDied","Data":"6c54d81f09eac5cd94131bbd39322eb9c766d194119f70a80d1a6c22ad4b6224"}
Nov 02 07:51:07.376280 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:07.375771    4898 scope.go:122] "RemoveContainer" containerID="4d083dbf6877c556b912ca32ecff68020f4ed489889ed3dadc5a6e8caf95a5f3"
Nov 02 07:51:07.385259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:07.385022    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5" event={"ID":"99f95feb-7b41-431b-bdf4-ed81c4a58119","Type":"ContainerStarted","Data":"71c3936fa79d7fa77d701ac20e609691ab33fee3ec2c22c8317a082a0b3a55c0"}
Nov 02 07:51:07.385259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:07.385077    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5" event={"ID":"99f95feb-7b41-431b-bdf4-ed81c4a58119","Type":"ContainerStarted","Data":"aa7296dddeff41d627003df56315e99eeefcf9949f4b85f3e4bce0a00756a01b"}
Nov 02 07:51:07.436097 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:07.436074    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"]
Nov 02 07:51:07.440046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:07.440023    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-eb64ad-openshift-default-56856d5c57-ngg2z"]
Nov 02 07:51:07.467815 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:07.467784    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" path="/var/lib/kubelet/pods/4e1d63fb-e287-42c3-8496-237e4556e5f4/volumes"
Nov 02 07:51:07.478409 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:07.478366    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5" podStartSLOduration=1.478354362 podStartE2EDuration="1.478354362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:07.476327881 +0000 UTC m=+1618.166654864" watchObservedRunningTime="2026-11-02 07:51:07.478354362 +0000 UTC m=+1618.168681324"
Nov 02 07:51:08.398552 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:08.398517    4898 generic.go:356] "Generic (PLEG): container finished" podID="99f95feb-7b41-431b-bdf4-ed81c4a58119" containerID="71c3936fa79d7fa77d701ac20e609691ab33fee3ec2c22c8317a082a0b3a55c0" exitCode=0
Nov 02 07:51:08.399030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:08.398610    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5" event={"ID":"99f95feb-7b41-431b-bdf4-ed81c4a58119","Type":"ContainerDied","Data":"71c3936fa79d7fa77d701ac20e609691ab33fee3ec2c22c8317a082a0b3a55c0"}
Nov 02 07:51:09.005223 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.005181    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"]
Nov 02 07:51:09.005957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.005904    4898 cpu_manager.go:414] "containerMap: removing container" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy"
Nov 02 07:51:09.005957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.005929    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy"
Nov 02 07:51:09.006151 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.005979    4898 cpu_manager.go:414] "containerMap: removing container" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy"
Nov 02 07:51:09.006151 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.005986    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy"
Nov 02 07:51:09.006257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.006168    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy"
Nov 02 07:51:09.006257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.006192    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1d63fb-e287-42c3-8496-237e4556e5f4" containerName="istio-proxy"
Nov 02 07:51:09.010168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.010146    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"
Nov 02 07:51:09.027631 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.027612    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3624d8-dockercfg-sgztb\""
Nov 02 07:51:09.035063 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.035038    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"]
Nov 02 07:51:09.039924 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.039903    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3624d8\""
Nov 02 07:51:09.048514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.048484    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csmvc\" (UniqueName: \"kubernetes.io/projected/95858c5b-0b54-4563-9b6e-05e241a67e10-kube-api-access-csmvc\") pod \"payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g\" (UID: \"95858c5b-0b54-4563-9b6e-05e241a67e10\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"
Nov 02 07:51:09.048658 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.048638    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95858c5b-0b54-4563-9b6e-05e241a67e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g\" (UID: \"95858c5b-0b54-4563-9b6e-05e241a67e10\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"
Nov 02 07:51:09.147951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.127763    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"]
Nov 02 07:51:09.191132 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.162929    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95858c5b-0b54-4563-9b6e-05e241a67e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g\" (UID: \"95858c5b-0b54-4563-9b6e-05e241a67e10\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"
Nov 02 07:51:09.191645 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.191616    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csmvc\" (UniqueName: \"kubernetes.io/projected/95858c5b-0b54-4563-9b6e-05e241a67e10-kube-api-access-csmvc\") pod \"payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g\" (UID: \"95858c5b-0b54-4563-9b6e-05e241a67e10\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"
Nov 02 07:51:09.192346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.175096    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95858c5b-0b54-4563-9b6e-05e241a67e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g\" (UID: \"95858c5b-0b54-4563-9b6e-05e241a67e10\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"
Nov 02 07:51:09.196318 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.196288    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"]
Nov 02 07:51:09.196454 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.196405    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"
Nov 02 07:51:09.291449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.291391    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csmvc\" (UniqueName: \"kubernetes.io/projected/95858c5b-0b54-4563-9b6e-05e241a67e10-kube-api-access-csmvc\") pod \"payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g\" (UID: \"95858c5b-0b54-4563-9b6e-05e241a67e10\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"
Nov 02 07:51:09.293764 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.293724    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7gtl\" (UniqueName: \"kubernetes.io/projected/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-kube-api-access-d7gtl\") pod \"payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm\" (UID: \"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"
Nov 02 07:51:09.293957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.293938    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm\" (UID: \"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"
Nov 02 07:51:09.324548 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.324515    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"
Nov 02 07:51:09.409501 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.409469    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm\" (UID: \"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"
Nov 02 07:51:09.409975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.409651    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7gtl\" (UniqueName: \"kubernetes.io/projected/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-kube-api-access-d7gtl\") pod \"payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm\" (UID: \"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"
Nov 02 07:51:09.410748 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.410722    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm\" (UID: \"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"
Nov 02 07:51:09.447087 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.447022    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7gtl\" (UniqueName: \"kubernetes.io/projected/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-kube-api-access-d7gtl\") pod \"payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm\" (UID: \"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"
Nov 02 07:51:09.585935 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.585806    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"
Nov 02 07:51:09.638618 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:51:09.638588    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95858c5b_0b54_4563_9b6e_05e241a67e10.slice/crio-6eca17f21ccdd0e63fb7b0dec18e37ae966ea9fbaf7f9f1a762bc635932a87cd WatchSource:0}: Error finding container 6eca17f21ccdd0e63fb7b0dec18e37ae966ea9fbaf7f9f1a762bc635932a87cd: Status 404 returned error can't find the container with id 6eca17f21ccdd0e63fb7b0dec18e37ae966ea9fbaf7f9f1a762bc635932a87cd
Nov 02 07:51:09.640789 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.640760    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"]
Nov 02 07:51:09.641870 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.641854    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:09.716383 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.716333    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f95feb-7b41-431b-bdf4-ed81c4a58119-kube-api-access-8l9qh\" (UniqueName: \"kubernetes.io/projected/99f95feb-7b41-431b-bdf4-ed81c4a58119-kube-api-access-8l9qh\") pod \"99f95feb-7b41-431b-bdf4-ed81c4a58119\" (UID: \"99f95feb-7b41-431b-bdf4-ed81c4a58119\") "
Nov 02 07:51:09.716692 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.716574    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-openshift-ca\") pod \"99f95feb-7b41-431b-bdf4-ed81c4a58119\" (UID: \"99f95feb-7b41-431b-bdf4-ed81c4a58119\") "
Nov 02 07:51:09.716692 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.716622    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-xks-ca\") pod \"99f95feb-7b41-431b-bdf4-ed81c4a58119\" (UID: \"99f95feb-7b41-431b-bdf4-ed81c4a58119\") "
Nov 02 07:51:09.718373 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.718326    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-openshift-ca" pod "99f95feb-7b41-431b-bdf4-ed81c4a58119" (UID: "99f95feb-7b41-431b-bdf4-ed81c4a58119"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:51:09.719793 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.719575    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-xks-ca" pod "99f95feb-7b41-431b-bdf4-ed81c4a58119" (UID: "99f95feb-7b41-431b-bdf4-ed81c4a58119"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:51:09.720933 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.720877    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f95feb-7b41-431b-bdf4-ed81c4a58119-kube-api-access-8l9qh" pod "99f95feb-7b41-431b-bdf4-ed81c4a58119" (UID: "99f95feb-7b41-431b-bdf4-ed81c4a58119"). InnerVolumeSpecName "kube-api-access-8l9qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:51:09.818296 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.818215    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:09.818296 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.818245    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f95feb-7b41-431b-bdf4-ed81c4a58119-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:09.818296 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.818259    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l9qh\" (UniqueName: \"kubernetes.io/projected/99f95feb-7b41-431b-bdf4-ed81c4a58119-kube-api-access-8l9qh\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:09.912193 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:51:09.912153    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dd73d7_b5ce_4ab6_916b_02dd18cbf177.slice/crio-79247ad723beaeb43cbe90e288356594e00adb49bb1201d4fe8a36da5e5ad4a9 WatchSource:0}: Error finding container 79247ad723beaeb43cbe90e288356594e00adb49bb1201d4fe8a36da5e5ad4a9: Status 404 returned error can't find the container with id 79247ad723beaeb43cbe90e288356594e00adb49bb1201d4fe8a36da5e5ad4a9
Nov 02 07:51:09.914124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:09.914101    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"]
Nov 02 07:51:10.426724 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:10.426689    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g" event={"ID":"95858c5b-0b54-4563-9b6e-05e241a67e10","Type":"ContainerStarted","Data":"f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0"}
Nov 02 07:51:10.427196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:10.426756    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"
Nov 02 07:51:10.427196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:10.426783    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g" event={"ID":"95858c5b-0b54-4563-9b6e-05e241a67e10","Type":"ContainerStarted","Data":"6eca17f21ccdd0e63fb7b0dec18e37ae966ea9fbaf7f9f1a762bc635932a87cd"}
Nov 02 07:51:10.428121 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:10.428100    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"
Nov 02 07:51:10.428256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:10.428151    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5" event={"ID":"99f95feb-7b41-431b-bdf4-ed81c4a58119","Type":"ContainerDied","Data":"aa7296dddeff41d627003df56315e99eeefcf9949f4b85f3e4bce0a00756a01b"}
Nov 02 07:51:10.428256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:10.428195    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa7296dddeff41d627003df56315e99eeefcf9949f4b85f3e4bce0a00756a01b"
Nov 02 07:51:10.430034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:10.430007    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm" event={"ID":"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177","Type":"ContainerStarted","Data":"2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f"}
Nov 02 07:51:10.430122 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:10.430058    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm" event={"ID":"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177","Type":"ContainerStarted","Data":"79247ad723beaeb43cbe90e288356594e00adb49bb1201d4fe8a36da5e5ad4a9"}
Nov 02 07:51:10.430122 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:10.430088    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"
Nov 02 07:51:10.470324 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:10.470274    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g" podStartSLOduration=2.470259729 podStartE2EDuration="2.470259729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:10.467141474 +0000 UTC m=+1621.157468438" watchObservedRunningTime="2026-11-02 07:51:10.470259729 +0000 UTC m=+1621.160586671"
Nov 02 07:51:10.525609 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:10.525559    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm" podStartSLOduration=1.525543775 podStartE2EDuration="1.525543775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:10.522845073 +0000 UTC m=+1621.213172049" watchObservedRunningTime="2026-11-02 07:51:10.525543775 +0000 UTC m=+1621.215870782"
Nov 02 07:51:11.235674 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:11.235635    4898 scope.go:122] "RemoveContainer" containerID="812e1dab2a4d367460238b96357e1ec9af08e4f88bbb93298a73c4157bff655b"
Nov 02 07:51:14.988871 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:51:14.988832    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:51:15.047986 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.047949    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"]
Nov 02 07:51:15.048664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.048649    4898 cpu_manager.go:414] "containerMap: removing container" podUID="99f95feb-7b41-431b-bdf4-ed81c4a58119" containerName="revoke-keys"
Nov 02 07:51:15.048729 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.048665    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f95feb-7b41-431b-bdf4-ed81c4a58119" containerName="revoke-keys"
Nov 02 07:51:15.048789 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.048747    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f95feb-7b41-431b-bdf4-ed81c4a58119" containerName="revoke-keys"
Nov 02 07:51:15.054608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.054587    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:15.076862 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.076840    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"]
Nov 02 07:51:15.084927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.084754    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:51:15.084927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.084770    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:51:15.084927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.084801    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:51:15.084927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.084866    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:51:15.194565 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.194513    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps2wb\" (UniqueName: \"kubernetes.io/projected/42e18c66-1de9-4087-9a95-91af68c032b3-kube-api-access-ps2wb\") pod \"maas-api-revoke-sub-cadd308b-mvvxn\" (UID: \"42e18c66-1de9-4087-9a95-91af68c032b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:15.194775 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.194682    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cadd308b-mvvxn\" (UID: \"42e18c66-1de9-4087-9a95-91af68c032b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:15.194775 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.194710    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cadd308b-mvvxn\" (UID: \"42e18c66-1de9-4087-9a95-91af68c032b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:15.296298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.296223    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cadd308b-mvvxn\" (UID: \"42e18c66-1de9-4087-9a95-91af68c032b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:15.296298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.296259    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cadd308b-mvvxn\" (UID: \"42e18c66-1de9-4087-9a95-91af68c032b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:15.296527 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.296400    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps2wb\" (UniqueName: \"kubernetes.io/projected/42e18c66-1de9-4087-9a95-91af68c032b3-kube-api-access-ps2wb\") pod \"maas-api-revoke-sub-cadd308b-mvvxn\" (UID: \"42e18c66-1de9-4087-9a95-91af68c032b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:15.297013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.296987    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cadd308b-mvvxn\" (UID: \"42e18c66-1de9-4087-9a95-91af68c032b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:15.298720 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.298695    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cadd308b-mvvxn\" (UID: \"42e18c66-1de9-4087-9a95-91af68c032b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:15.320570 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.320538    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps2wb\" (UniqueName: \"kubernetes.io/projected/42e18c66-1de9-4087-9a95-91af68c032b3-kube-api-access-ps2wb\") pod \"maas-api-revoke-sub-cadd308b-mvvxn\" (UID: \"42e18c66-1de9-4087-9a95-91af68c032b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:15.371433 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.371408    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:15.680183 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:51:15.680142    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e18c66_1de9_4087_9a95_91af68c032b3.slice/crio-9259b3cb59be065b9878367b72c6cb2d0413de32bf9c89c1ec74ad7c6ebdbe00 WatchSource:0}: Error finding container 9259b3cb59be065b9878367b72c6cb2d0413de32bf9c89c1ec74ad7c6ebdbe00: Status 404 returned error can't find the container with id 9259b3cb59be065b9878367b72c6cb2d0413de32bf9c89c1ec74ad7c6ebdbe00
Nov 02 07:51:15.681270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:15.681237    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"]
Nov 02 07:51:16.330969 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.330881    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"]
Nov 02 07:51:16.335398 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.335376    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.378909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.373502    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"]
Nov 02 07:51:16.382829 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.382803    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"]
Nov 02 07:51:16.387817 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.387793    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.396286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.396261    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"]
Nov 02 07:51:16.484274 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.484238    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn" event={"ID":"42e18c66-1de9-4087-9a95-91af68c032b3","Type":"ContainerStarted","Data":"cab198abef887c957d783a28a4a624637237efecd2909596d89c4c650ca9c321"}
Nov 02 07:51:16.484475 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.484298    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn" event={"ID":"42e18c66-1de9-4087-9a95-91af68c032b3","Type":"ContainerStarted","Data":"9259b3cb59be065b9878367b72c6cb2d0413de32bf9c89c1ec74ad7c6ebdbe00"}
Nov 02 07:51:16.512901 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.512861    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687d6399-45b2-48ea-9290-bf675a2b22bb-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.513043 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.512930    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5ntb\" (UniqueName: \"kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-kube-api-access-d5ntb\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.513043 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.512978    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.513173 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513078    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.513173 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513134    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.513262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513181    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.513262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513239    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.513387 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513371    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.513435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513408    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.513487 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513432    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.513487 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513452    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.513564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513494    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.513655 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513631    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.513708 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513695    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.513748 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513734    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.513807 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513791    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.513844 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513828    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn9vz\" (UniqueName: \"kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-kube-api-access-mn9vz\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.513943 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.513928    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.531444 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.529589    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn" podStartSLOduration=1.5295720259999999 podStartE2EDuration="1.529572026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:16.524595768 +0000 UTC m=+1627.214922732" watchObservedRunningTime="2026-11-02 07:51:16.529572026 +0000 UTC m=+1627.219898991"
Nov 02 07:51:16.615491 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.615462    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.615626 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.615503    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.615626 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.615527    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.615626 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.615556    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.615626 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.615578    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.615869 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.615704    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.615869 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.615785    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.615869 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.615813    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.615869 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.615847    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.616118 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.615882    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn9vz\" (UniqueName: \"kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-kube-api-access-mn9vz\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.616118 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.615948    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.616118 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.615997    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.616118 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.616050    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.616457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.616244    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.616457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.616246    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-data\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.616457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.616427    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.616457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.616445    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687d6399-45b2-48ea-9290-bf675a2b22bb-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.616668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.616478    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5ntb\" (UniqueName: \"kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-kube-api-access-d5ntb\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.616668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.616507    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.616668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.616537    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.616668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.616582    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.616668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.616632    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.616940 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.616684    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.617106 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.617076    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.617175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.617141    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-certs\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.617241 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.617185    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-credential-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.617319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.617287    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-socket\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.617473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.617440    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687d6399-45b2-48ea-9290-bf675a2b22bb-istiod-ca-cert\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.618431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.618402    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.619069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.619048    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.619285 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.619267    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-envoy\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.619647 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.619625    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-podinfo\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.649281 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.649258    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.651364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.651339    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-token\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.652031 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.652007    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5ntb\" (UniqueName: \"kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-kube-api-access-d5ntb\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:16.652418 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.652398    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn9vz\" (UniqueName: \"kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-kube-api-access-mn9vz\") pod \"e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") " pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.713549 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.713517    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:16.951254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:16.951221    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:17.055496 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.055467    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"]
Nov 02 07:51:17.058880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.058840    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:17.058986 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.058946    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:17.059046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.058994    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:17.271492 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:51:17.271461    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687d6399_45b2_48ea_9290_bf675a2b22bb.slice/crio-5af7c08aae6ee7a6148dad43e0441161e419a9e57cbd9aa425673bf8b428ecbc WatchSource:0}: Error finding container 5af7c08aae6ee7a6148dad43e0441161e419a9e57cbd9aa425673bf8b428ecbc: Status 404 returned error can't find the container with id 5af7c08aae6ee7a6148dad43e0441161e419a9e57cbd9aa425673bf8b428ecbc
Nov 02 07:51:17.272925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.272872    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"]
Nov 02 07:51:17.274606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.274571    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:17.274696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.274637    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:17.274696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.274671    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:17.496511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.496413    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss" event={"ID":"fd3bbf16-61be-4148-b7ff-5c862aed49d1","Type":"ContainerStarted","Data":"d424c7f29b35eacb4cc35208b15b06a35fbe91d494b30c73d84bae87a16b7fe4"}
Nov 02 07:51:17.496511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.496476    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss" event={"ID":"fd3bbf16-61be-4148-b7ff-5c862aed49d1","Type":"ContainerStarted","Data":"0bec73b8b2b683728bcf8c1718df52b6a3c3ed6850f0700787c14eecb576bb7f"}
Nov 02 07:51:17.499378 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.499299    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s" event={"ID":"687d6399-45b2-48ea-9290-bf675a2b22bb","Type":"ContainerStarted","Data":"a19d2637adbe3ed64a57472a9d9def34ab0878627922a2efae004a4fd996cdeb"}
Nov 02 07:51:17.499378 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.499347    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s" event={"ID":"687d6399-45b2-48ea-9290-bf675a2b22bb","Type":"ContainerStarted","Data":"5af7c08aae6ee7a6148dad43e0441161e419a9e57cbd9aa425673bf8b428ecbc"}
Nov 02 07:51:17.501551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.501513    4898 generic.go:356] "Generic (PLEG): container finished" podID="42e18c66-1de9-4087-9a95-91af68c032b3" containerID="cab198abef887c957d783a28a4a624637237efecd2909596d89c4c650ca9c321" exitCode=0
Nov 02 07:51:17.501677 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.501609    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn" event={"ID":"42e18c66-1de9-4087-9a95-91af68c032b3","Type":"ContainerDied","Data":"cab198abef887c957d783a28a4a624637237efecd2909596d89c4c650ca9c321"}
Nov 02 07:51:17.599967 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.596010    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss" podStartSLOduration=1.595986904 podStartE2EDuration="1.595986904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:17.546144474 +0000 UTC m=+1628.236471458" watchObservedRunningTime="2026-11-02 07:51:17.595986904 +0000 UTC m=+1628.286313868"
Nov 02 07:51:17.646735 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.646677    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s" podStartSLOduration=1.6466636590000001 podStartE2EDuration="1.646663659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:17.603221862 +0000 UTC m=+1628.293548826" watchObservedRunningTime="2026-11-02 07:51:17.646663659 +0000 UTC m=+1628.336990622"
Nov 02 07:51:17.713793 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.713748    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:17.951584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:17.951537    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:18.729912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:18.729175    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 07:51:18.729912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:18.729252    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss" podUID="fd3bbf16-61be-4148-b7ff-5c862aed49d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 07:51:18.942755 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:18.942730    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:18.953442 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:18.953414    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e18c66-1de9-4087-9a95-91af68c032b3-kube-api-access-ps2wb\" (UniqueName: \"kubernetes.io/projected/42e18c66-1de9-4087-9a95-91af68c032b3-kube-api-access-ps2wb\") pod \"42e18c66-1de9-4087-9a95-91af68c032b3\" (UID: \"42e18c66-1de9-4087-9a95-91af68c032b3\") "
Nov 02 07:51:18.953563 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:18.953493    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-openshift-ca\") pod \"42e18c66-1de9-4087-9a95-91af68c032b3\" (UID: \"42e18c66-1de9-4087-9a95-91af68c032b3\") "
Nov 02 07:51:18.953669 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:18.953655    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-xks-ca\") pod \"42e18c66-1de9-4087-9a95-91af68c032b3\" (UID: \"42e18c66-1de9-4087-9a95-91af68c032b3\") "
Nov 02 07:51:18.953848 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:18.953825    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 07:51:18.953884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:18.953869    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s" podUID="687d6399-45b2-48ea-9290-bf675a2b22bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 07:51:18.954282 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:18.954261    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-openshift-ca" pod "42e18c66-1de9-4087-9a95-91af68c032b3" (UID: "42e18c66-1de9-4087-9a95-91af68c032b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:51:18.980155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:18.977041    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-xks-ca" pod "42e18c66-1de9-4087-9a95-91af68c032b3" (UID: "42e18c66-1de9-4087-9a95-91af68c032b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:51:19.012913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:19.012082    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e18c66-1de9-4087-9a95-91af68c032b3-kube-api-access-ps2wb" pod "42e18c66-1de9-4087-9a95-91af68c032b3" (UID: "42e18c66-1de9-4087-9a95-91af68c032b3"). InnerVolumeSpecName "kube-api-access-ps2wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:51:19.097912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:19.078267    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps2wb\" (UniqueName: \"kubernetes.io/projected/42e18c66-1de9-4087-9a95-91af68c032b3-kube-api-access-ps2wb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:19.097912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:19.078306    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:19.097912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:19.078322    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42e18c66-1de9-4087-9a95-91af68c032b3-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:19.541551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:19.540971    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"
Nov 02 07:51:19.541551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:19.541014    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn" event={"ID":"42e18c66-1de9-4087-9a95-91af68c032b3","Type":"ContainerDied","Data":"9259b3cb59be065b9878367b72c6cb2d0413de32bf9c89c1ec74ad7c6ebdbe00"}
Nov 02 07:51:19.541551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:19.541062    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9259b3cb59be065b9878367b72c6cb2d0413de32bf9c89c1ec74ad7c6ebdbe00"
Nov 02 07:51:19.716604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:19.716564    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 07:51:19.716922 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:19.716866    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss" podUID="fd3bbf16-61be-4148-b7ff-5c862aed49d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 07:51:19.952101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:19.952069    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 07:51:19.952571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:19.952124    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s" podUID="687d6399-45b2-48ea-9290-bf675a2b22bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 07:51:20.714328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:20.714287    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 07:51:20.714611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:20.714583    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss" podUID="fd3bbf16-61be-4148-b7ff-5c862aed49d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 07:51:20.951550 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:20.951514    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 07:51:20.951738 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:20.951566    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s" podUID="687d6399-45b2-48ea-9290-bf675a2b22bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 07:51:21.441421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:21.441389    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"
Nov 02 07:51:21.442553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:21.442390    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"
Nov 02 07:51:21.743025 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:21.729924    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 07:51:21.743025 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:21.729985    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss" podUID="fd3bbf16-61be-4148-b7ff-5c862aed49d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 07:51:21.951856 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:21.951821    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 07:51:21.952039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:21.951880    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s" podUID="687d6399-45b2-48ea-9290-bf675a2b22bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 07:51:22.719316 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:22.719277    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 07:51:22.719944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:22.719882    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss" podUID="fd3bbf16-61be-4148-b7ff-5c862aed49d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 07:51:22.951784 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:22.951749    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 07:51:22.952012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:22.951823    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s" podUID="687d6399-45b2-48ea-9290-bf675a2b22bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 07:51:23.714264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:23.714221    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 02 07:51:23.714440 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:23.714291    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss" podUID="fd3bbf16-61be-4148-b7ff-5c862aed49d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 02 07:51:23.952434 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:23.952390    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 07:51:23.952812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:23.952460    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s" podUID="687d6399-45b2-48ea-9290-bf675a2b22bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 07:51:24.719207 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:24.719174    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:24.719652 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:24.719620    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:24.720507 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:24.720488    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:51:24.964068 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:24.964036    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:24.965865 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:24.965840    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:24.968092 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:24.968065    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:51:29.068915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.066950    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"]
Nov 02 07:51:29.068915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.068000    4898 cpu_manager.go:414] "containerMap: removing container" podUID="42e18c66-1de9-4087-9a95-91af68c032b3" containerName="revoke-keys"
Nov 02 07:51:29.068915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.068023    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e18c66-1de9-4087-9a95-91af68c032b3" containerName="revoke-keys"
Nov 02 07:51:29.068915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.068212    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e18c66-1de9-4087-9a95-91af68c032b3" containerName="revoke-keys"
Nov 02 07:51:29.100566 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.100531    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:29.131130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.131086    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"]
Nov 02 07:51:29.177954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.148864    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:51:29.177954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.149568    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:51:29.177954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.149659    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:51:29.177954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.149719    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:51:29.177954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.151602    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4svdb\" (UniqueName: \"kubernetes.io/projected/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-kube-api-access-4svdb\") pod \"maas-api-revoke-sub-02a0fc2d-slmw5\" (UID: \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:29.177954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.174214    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02a0fc2d-slmw5\" (UID: \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:29.177954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.174266    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02a0fc2d-slmw5\" (UID: \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:29.298920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.280469    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svdb\" (UniqueName: \"kubernetes.io/projected/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-kube-api-access-4svdb\") pod \"maas-api-revoke-sub-02a0fc2d-slmw5\" (UID: \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:29.298920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.280574    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02a0fc2d-slmw5\" (UID: \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:29.298920 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.280611    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02a0fc2d-slmw5\" (UID: \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:29.349974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.303764    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02a0fc2d-slmw5\" (UID: \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:29.807196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.807156    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02a0fc2d-slmw5\" (UID: \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:29.836915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:29.826864    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4svdb\" (UniqueName: \"kubernetes.io/projected/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-kube-api-access-4svdb\") pod \"maas-api-revoke-sub-02a0fc2d-slmw5\" (UID: \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:30.082975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:30.082606    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:30.100796 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:30.100747    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:30.131459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:30.129100    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:30.295912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:30.293058    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:51:30.295912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:30.293128    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:30.478917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:30.471194    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:30.503426 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:30.503377    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:30.585914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:30.577411    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:30.662138 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:30.661005    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 02 07:51:30.882970 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:30.873674    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 02 07:51:30.948777 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:30.945173    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 02 07:51:31.064916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:31.051875    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:31.064916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:31.061138    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:31.104981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:31.102521    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Nov 02 07:51:31.104981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:31.103180    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:51:31.615549 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:31.615493    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:31.766366 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:31.688436    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 02 07:51:31.935001 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:31.933555    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 02 07:51:31.936686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:31.936651    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:31.936951 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:31.936914    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:31.957247 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:31.957197    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:32.033588 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.033551    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:51:32.033861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.033843    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"]
Nov 02 07:51:32.034086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.034069    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:51:32.034319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.034225    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:51:32.034486 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.034471    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:51:32.034624 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.034612    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:51:32.034764 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.034751    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:51:32.078107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.078055    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:32.114130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.114102    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:51:32.115876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.115849    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:51:32.146979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.145595    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:51:32.147266 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.147236    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:51:32.148407 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.148386    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:51:32.166803 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.166769    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5" event={"ID":"1c1fe234-c577-4dbc-a3a8-2f66532e9be7","Type":"ContainerStarted","Data":"96a3bcecbbd54b27c59d5a3045bae0184ddf4baaf083e7fcb04556a05ac7009e"}
Nov 02 07:51:32.175491 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.174611    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:51:32.242798 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:32.225077    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:51:33.175327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:33.175286    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5" event={"ID":"1c1fe234-c577-4dbc-a3a8-2f66532e9be7","Type":"ContainerStarted","Data":"e657d5c3e534e22338e1f90d147ad23cb4773ccba237b377d3076b2aa2ed57b4"}
Nov 02 07:51:33.217431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:33.217368    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5" podStartSLOduration=5.217349821 podStartE2EDuration="5.217349821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:33.216008685 +0000 UTC m=+1643.906335668" watchObservedRunningTime="2026-11-02 07:51:33.217349821 +0000 UTC m=+1643.907676786"
Nov 02 07:51:34.186027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:34.185985    4898 generic.go:356] "Generic (PLEG): container finished" podID="1c1fe234-c577-4dbc-a3a8-2f66532e9be7" containerID="e657d5c3e534e22338e1f90d147ad23cb4773ccba237b377d3076b2aa2ed57b4" exitCode=0
Nov 02 07:51:34.186489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:34.186045    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5" event={"ID":"1c1fe234-c577-4dbc-a3a8-2f66532e9be7","Type":"ContainerDied","Data":"e657d5c3e534e22338e1f90d147ad23cb4773ccba237b377d3076b2aa2ed57b4"}
Nov 02 07:51:35.366431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:35.366401    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:35.371375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:35.371351    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-openshift-ca\") pod \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\" (UID: \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\") "
Nov 02 07:51:35.371506 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:35.371477    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-xks-ca\") pod \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\" (UID: \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\") "
Nov 02 07:51:35.371567 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:35.371529    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-kube-api-access-4svdb\" (UniqueName: \"kubernetes.io/projected/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-kube-api-access-4svdb\") pod \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\" (UID: \"1c1fe234-c577-4dbc-a3a8-2f66532e9be7\") "
Nov 02 07:51:35.371774 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:35.371743    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-openshift-ca" pod "1c1fe234-c577-4dbc-a3a8-2f66532e9be7" (UID: "1c1fe234-c577-4dbc-a3a8-2f66532e9be7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:51:35.372021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:35.372004    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:35.373625 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:35.373601    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-xks-ca" pod "1c1fe234-c577-4dbc-a3a8-2f66532e9be7" (UID: "1c1fe234-c577-4dbc-a3a8-2f66532e9be7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:51:35.373625 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:35.373615    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-kube-api-access-4svdb" pod "1c1fe234-c577-4dbc-a3a8-2f66532e9be7" (UID: "1c1fe234-c577-4dbc-a3a8-2f66532e9be7"). InnerVolumeSpecName "kube-api-access-4svdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:51:35.473199 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:35.473107    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4svdb\" (UniqueName: \"kubernetes.io/projected/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-kube-api-access-4svdb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:35.473199 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:35.473139    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1fe234-c577-4dbc-a3a8-2f66532e9be7-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:36.206491 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:36.206461    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"
Nov 02 07:51:36.206491 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:36.206485    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5" event={"ID":"1c1fe234-c577-4dbc-a3a8-2f66532e9be7","Type":"ContainerDied","Data":"96a3bcecbbd54b27c59d5a3045bae0184ddf4baaf083e7fcb04556a05ac7009e"}
Nov 02 07:51:36.206721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:36.206540    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96a3bcecbbd54b27c59d5a3045bae0184ddf4baaf083e7fcb04556a05ac7009e"
Nov 02 07:51:43.402909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.401745    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"]
Nov 02 07:51:43.402909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.402798    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1c1fe234-c577-4dbc-a3a8-2f66532e9be7" containerName="revoke-keys"
Nov 02 07:51:43.402909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.402822    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1fe234-c577-4dbc-a3a8-2f66532e9be7" containerName="revoke-keys"
Nov 02 07:51:43.403537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.403065    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1fe234-c577-4dbc-a3a8-2f66532e9be7" containerName="revoke-keys"
Nov 02 07:51:43.420595 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.419091    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:43.425946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.423207    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"]
Nov 02 07:51:43.439311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.435168    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:51:43.469203 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.463730    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:51:43.469203 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.463850    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:51:43.469203 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.463977    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:51:43.490999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.487755    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a32ed4f-fxhnk\" (UID: \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:43.490999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.487865    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gblzz\" (UniqueName: \"kubernetes.io/projected/1678db81-b5f4-4bfe-9fce-316f7104bd3a-kube-api-access-gblzz\") pod \"maas-api-revoke-sub-1a32ed4f-fxhnk\" (UID: \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:43.490999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.488090    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a32ed4f-fxhnk\" (UID: \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:43.591787 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.591745    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a32ed4f-fxhnk\" (UID: \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:43.592228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.592199    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a32ed4f-fxhnk\" (UID: \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:43.604045 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.604009    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gblzz\" (UniqueName: \"kubernetes.io/projected/1678db81-b5f4-4bfe-9fce-316f7104bd3a-kube-api-access-gblzz\") pod \"maas-api-revoke-sub-1a32ed4f-fxhnk\" (UID: \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:43.604802 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.600830    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a32ed4f-fxhnk\" (UID: \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:43.616477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.616444    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a32ed4f-fxhnk\" (UID: \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:43.649971 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.642854    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gblzz\" (UniqueName: \"kubernetes.io/projected/1678db81-b5f4-4bfe-9fce-316f7104bd3a-kube-api-access-gblzz\") pod \"maas-api-revoke-sub-1a32ed4f-fxhnk\" (UID: \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:43.803673 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:43.803319    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:44.803940 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:44.782209    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"]
Nov 02 07:51:45.123914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:45.105041    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:45.254252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:45.254205    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:45.256004 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:45.255968    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Nov 02 07:51:45.308061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:45.308017    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:51:45.308438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:45.308405    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:45.396377 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:51:45.396264    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:51:45.584531 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:45.584479    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:45.687700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:45.687652    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:45.712770 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:45.712727    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk" event={"ID":"1678db81-b5f4-4bfe-9fce-316f7104bd3a","Type":"ContainerStarted","Data":"95e4e054c09b75723b9334891300b56eb101db5b8869920288e62ea06c6ee6de"}
Nov 02 07:51:45.715043 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:45.715003    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:45.953978 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:45.952100    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 02 07:51:46.034048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.034001    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Nov 02 07:51:46.118524 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.115991    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:46.118524 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.116824    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:51:46.163021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.162977    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 02 07:51:46.200091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.200036    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:46.257070 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.256966    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:46.286948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.261862    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:51:46.577940 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.544993    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded"
Nov 02 07:51:46.634609 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.626029    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 02 07:51:46.635617 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.635585    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:51:46.706175 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.705073    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:46.706787 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.706749    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:51:46.750537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.750488    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:46.779584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.779549    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:51:46.964026 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.963972    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:46.985404 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:46.985369    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:51:47.026034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.025997    4898 generic.go:356] "Generic (PLEG): container finished" podID="1678db81-b5f4-4bfe-9fce-316f7104bd3a" containerID="96db976464426ba4bf02a2fc99c2444aa2b24e78dbe3e62b6b24286b34740e09" exitCode=0
Nov 02 07:51:47.028289 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.028123    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk" event={"ID":"1678db81-b5f4-4bfe-9fce-316f7104bd3a","Type":"ContainerDied","Data":"96db976464426ba4bf02a2fc99c2444aa2b24e78dbe3e62b6b24286b34740e09"}
Nov 02 07:51:47.042052 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.042009    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:47.042787 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.042760    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:51:47.118099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.118056    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:47.191953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.168336    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:47.191953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.168877    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:51:47.227655 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.227557    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:47.243944 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.236584    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:51:47.353913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.257583    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:51:47.525975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.505653    4898 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Nov 02 07:51:47.525975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.506232    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 07:51:47.571125 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.571089    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 07:51:47.571436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.571418    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 07:51:47.577576 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.577552    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:51:47.580379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.580359    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"]
Nov 02 07:51:47.613153 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.613122    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 07:51:47.613401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.613387    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 07:51:47.614189 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.614166    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:47.621928 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.619101    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"]
Nov 02 07:51:47.639416 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.639389    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:51:47.639660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.639642    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 07:51:47.639881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.639863    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:51:47.720574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.719607    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc8z5\" (UniqueName: \"kubernetes.io/projected/05a7e857-1289-4d52-8b85-c7af88c061ed-kube-api-access-cc8z5\") pod \"maas-api-revoke-sub-b33391f3-hqn2n\" (UID: \"05a7e857-1289-4d52-8b85-c7af88c061ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:47.720574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.719737    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b33391f3-hqn2n\" (UID: \"05a7e857-1289-4d52-8b85-c7af88c061ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:47.720574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.719801    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b33391f3-hqn2n\" (UID: \"05a7e857-1289-4d52-8b85-c7af88c061ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:47.821811 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.821703    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc8z5\" (UniqueName: \"kubernetes.io/projected/05a7e857-1289-4d52-8b85-c7af88c061ed-kube-api-access-cc8z5\") pod \"maas-api-revoke-sub-b33391f3-hqn2n\" (UID: \"05a7e857-1289-4d52-8b85-c7af88c061ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:47.822489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.822462    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b33391f3-hqn2n\" (UID: \"05a7e857-1289-4d52-8b85-c7af88c061ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:47.822698 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.822678    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b33391f3-hqn2n\" (UID: \"05a7e857-1289-4d52-8b85-c7af88c061ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:47.824039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.823988    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b33391f3-hqn2n\" (UID: \"05a7e857-1289-4d52-8b85-c7af88c061ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:47.829411 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.829391    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b33391f3-hqn2n\" (UID: \"05a7e857-1289-4d52-8b85-c7af88c061ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:47.849946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:47.848153    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc8z5\" (UniqueName: \"kubernetes.io/projected/05a7e857-1289-4d52-8b85-c7af88c061ed-kube-api-access-cc8z5\") pod \"maas-api-revoke-sub-b33391f3-hqn2n\" (UID: \"05a7e857-1289-4d52-8b85-c7af88c061ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:48.021019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:48.020989    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:48.504853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:48.503596    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:48.598677 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:48.598644    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"]
Nov 02 07:51:48.644233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:48.644206    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1678db81-b5f4-4bfe-9fce-316f7104bd3a-kube-api-access-gblzz\" (UniqueName: \"kubernetes.io/projected/1678db81-b5f4-4bfe-9fce-316f7104bd3a-kube-api-access-gblzz\") pod \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\" (UID: \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\") "
Nov 02 07:51:48.644374 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:48.644254    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-xks-ca\") pod \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\" (UID: \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\") "
Nov 02 07:51:48.644374 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:48.644315    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-openshift-ca\") pod \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\" (UID: \"1678db81-b5f4-4bfe-9fce-316f7104bd3a\") "
Nov 02 07:51:48.645269 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:48.644735    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-openshift-ca" pod "1678db81-b5f4-4bfe-9fce-316f7104bd3a" (UID: "1678db81-b5f4-4bfe-9fce-316f7104bd3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:51:48.645269 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:48.645101    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:48.647217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:48.647195    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1678db81-b5f4-4bfe-9fce-316f7104bd3a-kube-api-access-gblzz" pod "1678db81-b5f4-4bfe-9fce-316f7104bd3a" (UID: "1678db81-b5f4-4bfe-9fce-316f7104bd3a"). InnerVolumeSpecName "kube-api-access-gblzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:51:48.650458 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:48.650436    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-xks-ca" pod "1678db81-b5f4-4bfe-9fce-316f7104bd3a" (UID: "1678db81-b5f4-4bfe-9fce-316f7104bd3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:51:48.746675 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:48.746589    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gblzz\" (UniqueName: \"kubernetes.io/projected/1678db81-b5f4-4bfe-9fce-316f7104bd3a-kube-api-access-gblzz\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:48.746675 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:48.746629    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1678db81-b5f4-4bfe-9fce-316f7104bd3a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:49.045671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:49.045586    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"
Nov 02 07:51:49.045671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:49.045637    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk" event={"ID":"1678db81-b5f4-4bfe-9fce-316f7104bd3a","Type":"ContainerDied","Data":"95e4e054c09b75723b9334891300b56eb101db5b8869920288e62ea06c6ee6de"}
Nov 02 07:51:49.046258 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:49.045689    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95e4e054c09b75723b9334891300b56eb101db5b8869920288e62ea06c6ee6de"
Nov 02 07:51:49.047435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:49.047402    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n" event={"ID":"05a7e857-1289-4d52-8b85-c7af88c061ed","Type":"ContainerStarted","Data":"9b9758aa90ec62c8127a308309a9348c06d74d57309141e922e73953bdaca688"}
Nov 02 07:51:49.047586 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:49.047441    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n" event={"ID":"05a7e857-1289-4d52-8b85-c7af88c061ed","Type":"ContainerStarted","Data":"8b46f4286c5c75031439b38f1cd4262d70ac7fdd3cfffe95656fae20ae89ed63"}
Nov 02 07:51:49.107051 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:49.106990    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n" podStartSLOduration=2.106975621 podStartE2EDuration="2.106975621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:51:49.09231532 +0000 UTC m=+1659.782642310" watchObservedRunningTime="2026-11-02 07:51:49.106975621 +0000 UTC m=+1659.797302658"
Nov 02 07:51:50.068981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:50.064401    4898 generic.go:356] "Generic (PLEG): container finished" podID="05a7e857-1289-4d52-8b85-c7af88c061ed" containerID="9b9758aa90ec62c8127a308309a9348c06d74d57309141e922e73953bdaca688" exitCode=0
Nov 02 07:51:50.068981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:50.064719    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n" event={"ID":"05a7e857-1289-4d52-8b85-c7af88c061ed","Type":"ContainerDied","Data":"9b9758aa90ec62c8127a308309a9348c06d74d57309141e922e73953bdaca688"}
Nov 02 07:51:51.341525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:51.341494    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:51.445200 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:51.445170    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-openshift-ca\") pod \"05a7e857-1289-4d52-8b85-c7af88c061ed\" (UID: \"05a7e857-1289-4d52-8b85-c7af88c061ed\") "
Nov 02 07:51:51.445369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:51.445216    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a7e857-1289-4d52-8b85-c7af88c061ed-kube-api-access-cc8z5\" (UniqueName: \"kubernetes.io/projected/05a7e857-1289-4d52-8b85-c7af88c061ed-kube-api-access-cc8z5\") pod \"05a7e857-1289-4d52-8b85-c7af88c061ed\" (UID: \"05a7e857-1289-4d52-8b85-c7af88c061ed\") "
Nov 02 07:51:51.445369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:51.445271    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-xks-ca\") pod \"05a7e857-1289-4d52-8b85-c7af88c061ed\" (UID: \"05a7e857-1289-4d52-8b85-c7af88c061ed\") "
Nov 02 07:51:51.445618 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:51.445591    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-openshift-ca" pod "05a7e857-1289-4d52-8b85-c7af88c061ed" (UID: "05a7e857-1289-4d52-8b85-c7af88c061ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:51:51.445835 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:51.445818    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:51.447498 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:51.447473    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-xks-ca" pod "05a7e857-1289-4d52-8b85-c7af88c061ed" (UID: "05a7e857-1289-4d52-8b85-c7af88c061ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:51:51.447606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:51.447532    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a7e857-1289-4d52-8b85-c7af88c061ed-kube-api-access-cc8z5" pod "05a7e857-1289-4d52-8b85-c7af88c061ed" (UID: "05a7e857-1289-4d52-8b85-c7af88c061ed"). InnerVolumeSpecName "kube-api-access-cc8z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:51:51.547742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:51.547706    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc8z5\" (UniqueName: \"kubernetes.io/projected/05a7e857-1289-4d52-8b85-c7af88c061ed-kube-api-access-cc8z5\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:51.547742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:51.547737    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a7e857-1289-4d52-8b85-c7af88c061ed-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:51:52.098420 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:52.098317    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"
Nov 02 07:51:52.098578 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:52.098334    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n" event={"ID":"05a7e857-1289-4d52-8b85-c7af88c061ed","Type":"ContainerDied","Data":"8b46f4286c5c75031439b38f1cd4262d70ac7fdd3cfffe95656fae20ae89ed63"}
Nov 02 07:51:52.098578 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:52.098458    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b46f4286c5c75031439b38f1cd4262d70ac7fdd3cfffe95656fae20ae89ed63"
Nov 02 07:51:56.135281 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:56.135248    4898 generic.go:356] "Generic (PLEG): container finished" podID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerID="663ae3b37b0b77c21928ec2af6d43b955fd884efb9f0c5cbef4654b460da4987" exitCode=0
Nov 02 07:51:56.135713 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:56.135320    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" event={"ID":"76acab59-e07d-49ae-9075-0accdbe6e5b1","Type":"ContainerDied","Data":"663ae3b37b0b77c21928ec2af6d43b955fd884efb9f0c5cbef4654b460da4987"}
Nov 02 07:51:56.135817 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:56.135800    4898 scope.go:122] "RemoveContainer" containerID="663ae3b37b0b77c21928ec2af6d43b955fd884efb9f0c5cbef4654b460da4987"
Nov 02 07:51:56.137336 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:56.137302    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:56.137430 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:56.137375    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:56.137430 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:56.137410    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:51:57.145150 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:57.145114    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" event={"ID":"76acab59-e07d-49ae-9075-0accdbe6e5b1","Type":"ContainerStarted","Data":"e9b07d639b6bf3f4ab753cc1550bc066016e85787909831fdbd881296682667e"}
Nov 02 07:51:57.310498 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:57.310467    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:51:57.310686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:57.310651    4898 patch_prober.go:29] interesting pod/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:51:57.310734 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:57.310707    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 07:51:58.310163 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:58.310126    4898 patch_prober.go:29] interesting pod/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:51:58.310576 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:58.310189    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 07:51:59.310302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:59.310262    4898 patch_prober.go:29] interesting pod/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:51:59.310744 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:51:59.310325    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 07:52:00.129751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.129718    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"]
Nov 02 07:52:00.130356 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.130331    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1678db81-b5f4-4bfe-9fce-316f7104bd3a" containerName="revoke-keys"
Nov 02 07:52:00.130356 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.130360    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1678db81-b5f4-4bfe-9fce-316f7104bd3a" containerName="revoke-keys"
Nov 02 07:52:00.130452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.130402    4898 cpu_manager.go:414] "containerMap: removing container" podUID="05a7e857-1289-4d52-8b85-c7af88c061ed" containerName="revoke-keys"
Nov 02 07:52:00.130452 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.130408    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a7e857-1289-4d52-8b85-c7af88c061ed" containerName="revoke-keys"
Nov 02 07:52:00.130530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.130520    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1678db81-b5f4-4bfe-9fce-316f7104bd3a" containerName="revoke-keys"
Nov 02 07:52:00.130566 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.130539    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a7e857-1289-4d52-8b85-c7af88c061ed" containerName="revoke-keys"
Nov 02 07:52:00.134479 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.134458    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:00.152472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.152447    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"]
Nov 02 07:52:00.179929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.161074    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:52:00.179929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.161164    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:52:00.179929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.166101    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:52:00.179929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.166219    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:52:00.206038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.206003    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbnpl\" (UniqueName: \"kubernetes.io/projected/e003706c-d239-45f3-8918-a6db4b2505f9-kube-api-access-cbnpl\") pod \"maas-api-revoke-sub-c0182346-9zjv6\" (UID: \"e003706c-d239-45f3-8918-a6db4b2505f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:00.206244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.206140    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0182346-9zjv6\" (UID: \"e003706c-d239-45f3-8918-a6db4b2505f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:00.206244 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.206177    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0182346-9zjv6\" (UID: \"e003706c-d239-45f3-8918-a6db4b2505f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:00.307201 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.307161    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbnpl\" (UniqueName: \"kubernetes.io/projected/e003706c-d239-45f3-8918-a6db4b2505f9-kube-api-access-cbnpl\") pod \"maas-api-revoke-sub-c0182346-9zjv6\" (UID: \"e003706c-d239-45f3-8918-a6db4b2505f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:00.307400 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.307266    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0182346-9zjv6\" (UID: \"e003706c-d239-45f3-8918-a6db4b2505f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:00.307400 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.307294    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0182346-9zjv6\" (UID: \"e003706c-d239-45f3-8918-a6db4b2505f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:00.307993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.307968    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0182346-9zjv6\" (UID: \"e003706c-d239-45f3-8918-a6db4b2505f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:00.309959 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.309925    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0182346-9zjv6\" (UID: \"e003706c-d239-45f3-8918-a6db4b2505f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:00.310066 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.309952    4898 patch_prober.go:29] interesting pod/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:52:00.310066 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.310004    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 07:52:00.330826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.330801    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbnpl\" (UniqueName: \"kubernetes.io/projected/e003706c-d239-45f3-8918-a6db4b2505f9-kube-api-access-cbnpl\") pod \"maas-api-revoke-sub-c0182346-9zjv6\" (UID: \"e003706c-d239-45f3-8918-a6db4b2505f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:00.449138 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.449107    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:00.788608 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:52:00.788572    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode003706c_d239_45f3_8918_a6db4b2505f9.slice/crio-a71563cf8632ce13e5df958d3b09a64c7fa7b48d2f2b3b2e0bcc6bbd10f45cf0 WatchSource:0}: Error finding container a71563cf8632ce13e5df958d3b09a64c7fa7b48d2f2b3b2e0bcc6bbd10f45cf0: Status 404 returned error can't find the container with id a71563cf8632ce13e5df958d3b09a64c7fa7b48d2f2b3b2e0bcc6bbd10f45cf0
Nov 02 07:52:00.790343 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:00.790227    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"]
Nov 02 07:52:01.214717 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.214672    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6" event={"ID":"e003706c-d239-45f3-8918-a6db4b2505f9","Type":"ContainerStarted","Data":"a71563cf8632ce13e5df958d3b09a64c7fa7b48d2f2b3b2e0bcc6bbd10f45cf0"}
Nov 02 07:52:01.310230 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.310194    4898 patch_prober.go:29] interesting pod/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 07:52:01.310499 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.310471    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 07:52:01.608520 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.608483    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"]
Nov 02 07:52:01.612727 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.612704    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:01.629331 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.629296    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2vfn\" (UniqueName: \"kubernetes.io/projected/66a2055a-e183-410d-9c16-2cc678f20f8e-kube-api-access-p2vfn\") pod \"maas-api-revoke-sub-4d28f2e0-r9gwk\" (UID: \"66a2055a-e183-410d-9c16-2cc678f20f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:01.629542 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.629521    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d28f2e0-r9gwk\" (UID: \"66a2055a-e183-410d-9c16-2cc678f20f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:01.629633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.629561    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d28f2e0-r9gwk\" (UID: \"66a2055a-e183-410d-9c16-2cc678f20f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:01.631738 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.631711    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"]
Nov 02 07:52:01.731064 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.731024    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d28f2e0-r9gwk\" (UID: \"66a2055a-e183-410d-9c16-2cc678f20f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:01.731358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.731092    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d28f2e0-r9gwk\" (UID: \"66a2055a-e183-410d-9c16-2cc678f20f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:01.731358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.731173    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2vfn\" (UniqueName: \"kubernetes.io/projected/66a2055a-e183-410d-9c16-2cc678f20f8e-kube-api-access-p2vfn\") pod \"maas-api-revoke-sub-4d28f2e0-r9gwk\" (UID: \"66a2055a-e183-410d-9c16-2cc678f20f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:01.732051 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.732028    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d28f2e0-r9gwk\" (UID: \"66a2055a-e183-410d-9c16-2cc678f20f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:01.734425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.734400    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d28f2e0-r9gwk\" (UID: \"66a2055a-e183-410d-9c16-2cc678f20f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:01.758918 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.755419    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2vfn\" (UniqueName: \"kubernetes.io/projected/66a2055a-e183-410d-9c16-2cc678f20f8e-kube-api-access-p2vfn\") pod \"maas-api-revoke-sub-4d28f2e0-r9gwk\" (UID: \"66a2055a-e183-410d-9c16-2cc678f20f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:01.928788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:01.928753    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:02.226132 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:02.226094    4898 generic.go:356] "Generic (PLEG): container finished" podID="e003706c-d239-45f3-8918-a6db4b2505f9" containerID="9ccf854d3bf619bdcf6ad8c88a7750c00fcbd5cb9d1ad9d69c448122bc316a9c" exitCode=0
Nov 02 07:52:02.226326 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:02.226190    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6" event={"ID":"e003706c-d239-45f3-8918-a6db4b2505f9","Type":"ContainerDied","Data":"9ccf854d3bf619bdcf6ad8c88a7750c00fcbd5cb9d1ad9d69c448122bc316a9c"}
Nov 02 07:52:02.276977 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:52:02.276936    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a2055a_e183_410d_9c16_2cc678f20f8e.slice/crio-793800c5e19601973d5c75e54959825cfa63dc07242d5a8ac24f467a45a053d8 WatchSource:0}: Error finding container 793800c5e19601973d5c75e54959825cfa63dc07242d5a8ac24f467a45a053d8: Status 404 returned error can't find the container with id 793800c5e19601973d5c75e54959825cfa63dc07242d5a8ac24f467a45a053d8
Nov 02 07:52:02.283967 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:02.283911    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"]
Nov 02 07:52:02.394088 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:02.394041    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:52:02.394525 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:02.394470    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:52:02.395769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:02.395508    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:52:03.236202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.236159    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk" event={"ID":"66a2055a-e183-410d-9c16-2cc678f20f8e","Type":"ContainerStarted","Data":"266135aaf286b28731679f610ffffa2068afcfe9fbc329cc8b77ba982231c957"}
Nov 02 07:52:03.236676 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.236221    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk" event={"ID":"66a2055a-e183-410d-9c16-2cc678f20f8e","Type":"ContainerStarted","Data":"793800c5e19601973d5c75e54959825cfa63dc07242d5a8ac24f467a45a053d8"}
Nov 02 07:52:03.281028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.280973    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk" podStartSLOduration=2.28095882 podStartE2EDuration="2.28095882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:03.280420628 +0000 UTC m=+1673.970747593" watchObservedRunningTime="2026-11-02 07:52:03.28095882 +0000 UTC m=+1673.971285785"
Nov 02 07:52:03.416587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.416556    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:03.455872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.455835    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-xks-ca\") pod \"e003706c-d239-45f3-8918-a6db4b2505f9\" (UID: \"e003706c-d239-45f3-8918-a6db4b2505f9\") "
Nov 02 07:52:03.456164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.456135    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e003706c-d239-45f3-8918-a6db4b2505f9-kube-api-access-cbnpl\" (UniqueName: \"kubernetes.io/projected/e003706c-d239-45f3-8918-a6db4b2505f9-kube-api-access-cbnpl\") pod \"e003706c-d239-45f3-8918-a6db4b2505f9\" (UID: \"e003706c-d239-45f3-8918-a6db4b2505f9\") "
Nov 02 07:52:03.458151 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.457563    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-openshift-ca\") pod \"e003706c-d239-45f3-8918-a6db4b2505f9\" (UID: \"e003706c-d239-45f3-8918-a6db4b2505f9\") "
Nov 02 07:52:03.459458 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.459423    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-openshift-ca" pod "e003706c-d239-45f3-8918-a6db4b2505f9" (UID: "e003706c-d239-45f3-8918-a6db4b2505f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:52:03.459458 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.459431    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-xks-ca" pod "e003706c-d239-45f3-8918-a6db4b2505f9" (UID: "e003706c-d239-45f3-8918-a6db4b2505f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:52:03.471226 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.471196    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e003706c-d239-45f3-8918-a6db4b2505f9-kube-api-access-cbnpl" pod "e003706c-d239-45f3-8918-a6db4b2505f9" (UID: "e003706c-d239-45f3-8918-a6db4b2505f9"). InnerVolumeSpecName "kube-api-access-cbnpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:03.559748 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.559701    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:03.560012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.559985    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbnpl\" (UniqueName: \"kubernetes.io/projected/e003706c-d239-45f3-8918-a6db4b2505f9-kube-api-access-cbnpl\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:03.560012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:03.560011    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e003706c-d239-45f3-8918-a6db4b2505f9-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:04.246519 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:04.246487    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"
Nov 02 07:52:04.247060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:04.246525    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6" event={"ID":"e003706c-d239-45f3-8918-a6db4b2505f9","Type":"ContainerDied","Data":"a71563cf8632ce13e5df958d3b09a64c7fa7b48d2f2b3b2e0bcc6bbd10f45cf0"}
Nov 02 07:52:04.247060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:04.246575    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a71563cf8632ce13e5df958d3b09a64c7fa7b48d2f2b3b2e0bcc6bbd10f45cf0"
Nov 02 07:52:04.248263 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:04.248235    4898 generic.go:356] "Generic (PLEG): container finished" podID="66a2055a-e183-410d-9c16-2cc678f20f8e" containerID="266135aaf286b28731679f610ffffa2068afcfe9fbc329cc8b77ba982231c957" exitCode=0
Nov 02 07:52:04.248413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:04.248297    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk" event={"ID":"66a2055a-e183-410d-9c16-2cc678f20f8e","Type":"ContainerDied","Data":"266135aaf286b28731679f610ffffa2068afcfe9fbc329cc8b77ba982231c957"}
Nov 02 07:52:05.060911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.054917    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"]
Nov 02 07:52:05.060911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.055905    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e003706c-d239-45f3-8918-a6db4b2505f9" containerName="revoke-keys"
Nov 02 07:52:05.060911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.055926    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e003706c-d239-45f3-8918-a6db4b2505f9" containerName="revoke-keys"
Nov 02 07:52:05.060911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.056114    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e003706c-d239-45f3-8918-a6db4b2505f9" containerName="revoke-keys"
Nov 02 07:52:05.106910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.106635    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"
Nov 02 07:52:05.165913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.158143    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9ab4d2\""
Nov 02 07:52:05.188722 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.170136    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9ab4d2-dockercfg-x82xt\""
Nov 02 07:52:05.235954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.230865    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"]
Nov 02 07:52:05.271872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.271783    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d429cbf5-924c-4ba9-91dd-c0b877835f24-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq\" (UID: \"d429cbf5-924c-4ba9-91dd-c0b877835f24\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"
Nov 02 07:52:05.272358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.271969    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jv8m\" (UniqueName: \"kubernetes.io/projected/d429cbf5-924c-4ba9-91dd-c0b877835f24-kube-api-access-7jv8m\") pod \"payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq\" (UID: \"d429cbf5-924c-4ba9-91dd-c0b877835f24\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"
Nov 02 07:52:05.374136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.374085    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d429cbf5-924c-4ba9-91dd-c0b877835f24-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq\" (UID: \"d429cbf5-924c-4ba9-91dd-c0b877835f24\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"
Nov 02 07:52:05.374344 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.374203    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jv8m\" (UniqueName: \"kubernetes.io/projected/d429cbf5-924c-4ba9-91dd-c0b877835f24-kube-api-access-7jv8m\") pod \"payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq\" (UID: \"d429cbf5-924c-4ba9-91dd-c0b877835f24\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"
Nov 02 07:52:05.375073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.375042    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d429cbf5-924c-4ba9-91dd-c0b877835f24-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq\" (UID: \"d429cbf5-924c-4ba9-91dd-c0b877835f24\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"
Nov 02 07:52:05.398362 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.398326    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jv8m\" (UniqueName: \"kubernetes.io/projected/d429cbf5-924c-4ba9-91dd-c0b877835f24-kube-api-access-7jv8m\") pod \"payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq\" (UID: \"d429cbf5-924c-4ba9-91dd-c0b877835f24\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"
Nov 02 07:52:05.426079 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.426053    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:05.543089 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.543052    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"
Nov 02 07:52:05.577429 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.577339    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-xks-ca\") pod \"66a2055a-e183-410d-9c16-2cc678f20f8e\" (UID: \"66a2055a-e183-410d-9c16-2cc678f20f8e\") "
Nov 02 07:52:05.577592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.577518    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-openshift-ca\") pod \"66a2055a-e183-410d-9c16-2cc678f20f8e\" (UID: \"66a2055a-e183-410d-9c16-2cc678f20f8e\") "
Nov 02 07:52:05.577592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.577577    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a2055a-e183-410d-9c16-2cc678f20f8e-kube-api-access-p2vfn\" (UniqueName: \"kubernetes.io/projected/66a2055a-e183-410d-9c16-2cc678f20f8e-kube-api-access-p2vfn\") pod \"66a2055a-e183-410d-9c16-2cc678f20f8e\" (UID: \"66a2055a-e183-410d-9c16-2cc678f20f8e\") "
Nov 02 07:52:05.579041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.579012    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-openshift-ca" pod "66a2055a-e183-410d-9c16-2cc678f20f8e" (UID: "66a2055a-e183-410d-9c16-2cc678f20f8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:52:05.581157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.581117    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a2055a-e183-410d-9c16-2cc678f20f8e-kube-api-access-p2vfn" pod "66a2055a-e183-410d-9c16-2cc678f20f8e" (UID: "66a2055a-e183-410d-9c16-2cc678f20f8e"). InnerVolumeSpecName "kube-api-access-p2vfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:05.581357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.581331    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-xks-ca" pod "66a2055a-e183-410d-9c16-2cc678f20f8e" (UID: "66a2055a-e183-410d-9c16-2cc678f20f8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:52:05.692305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.682143    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:05.692305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.682178    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a2055a-e183-410d-9c16-2cc678f20f8e-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:05.692305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.682194    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2vfn\" (UniqueName: \"kubernetes.io/projected/66a2055a-e183-410d-9c16-2cc678f20f8e-kube-api-access-p2vfn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:05.891619 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:52:05.891588    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd429cbf5_924c_4ba9_91dd_c0b877835f24.slice/crio-3987bc7e37f2ea4ec9f9fbdcaba2bc460b8675d652e71f574dc3ee4f43d57a09 WatchSource:0}: Error finding container 3987bc7e37f2ea4ec9f9fbdcaba2bc460b8675d652e71f574dc3ee4f43d57a09: Status 404 returned error can't find the container with id 3987bc7e37f2ea4ec9f9fbdcaba2bc460b8675d652e71f574dc3ee4f43d57a09
Nov 02 07:52:05.896364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:05.896324    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"]
Nov 02 07:52:06.271076 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:06.271038    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq" event={"ID":"d429cbf5-924c-4ba9-91dd-c0b877835f24","Type":"ContainerStarted","Data":"e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458"}
Nov 02 07:52:06.271255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:06.271106    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq" event={"ID":"d429cbf5-924c-4ba9-91dd-c0b877835f24","Type":"ContainerStarted","Data":"3987bc7e37f2ea4ec9f9fbdcaba2bc460b8675d652e71f574dc3ee4f43d57a09"}
Nov 02 07:52:06.271255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:06.271130    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"
Nov 02 07:52:06.273270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:06.273238    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"
Nov 02 07:52:06.273688 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:06.273285    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk" event={"ID":"66a2055a-e183-410d-9c16-2cc678f20f8e","Type":"ContainerDied","Data":"793800c5e19601973d5c75e54959825cfa63dc07242d5a8ac24f467a45a053d8"}
Nov 02 07:52:06.273688 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:06.273340    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="793800c5e19601973d5c75e54959825cfa63dc07242d5a8ac24f467a45a053d8"
Nov 02 07:52:06.314576 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:06.314521    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq" podStartSLOduration=2.314507562 podStartE2EDuration="2.314507562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:06.312309858 +0000 UTC m=+1677.002636822" watchObservedRunningTime="2026-11-02 07:52:06.314507562 +0000 UTC m=+1677.004834526"
Nov 02 07:52:12.452512 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.452476    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"]
Nov 02 07:52:12.453159 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.453142    4898 cpu_manager.go:414] "containerMap: removing container" podUID="66a2055a-e183-410d-9c16-2cc678f20f8e" containerName="revoke-keys"
Nov 02 07:52:12.453202 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.453160    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a2055a-e183-410d-9c16-2cc678f20f8e" containerName="revoke-keys"
Nov 02 07:52:12.453301 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.453291    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a2055a-e183-410d-9c16-2cc678f20f8e" containerName="revoke-keys"
Nov 02 07:52:12.458081 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.458061    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:12.476358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.476330    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"]
Nov 02 07:52:12.482685 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.482655    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:52:12.482850 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.482734    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:52:12.482941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.482830    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:52:12.484755 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.484730    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:52:12.488825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.488796    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpt4s\" (UniqueName: \"kubernetes.io/projected/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-kube-api-access-cpt4s\") pod \"maas-api-revoke-sub-aeffbe7f-g86hn\" (UID: \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:12.488976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.488950    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeffbe7f-g86hn\" (UID: \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:12.489065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.489048    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeffbe7f-g86hn\" (UID: \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:12.590064 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.590024    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeffbe7f-g86hn\" (UID: \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:12.590231 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.590126    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeffbe7f-g86hn\" (UID: \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:12.590376 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.590320    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpt4s\" (UniqueName: \"kubernetes.io/projected/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-kube-api-access-cpt4s\") pod \"maas-api-revoke-sub-aeffbe7f-g86hn\" (UID: \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:12.590849 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.590818    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeffbe7f-g86hn\" (UID: \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:12.592797 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.592768    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeffbe7f-g86hn\" (UID: \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:12.613544 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.613520    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpt4s\" (UniqueName: \"kubernetes.io/projected/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-kube-api-access-cpt4s\") pod \"maas-api-revoke-sub-aeffbe7f-g86hn\" (UID: \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:12.777146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:12.777064    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:13.080042 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:52:13.080013    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d20daab_f0d4_4f9c_ac1d_70bca6c37618.slice/crio-e9771d55e062f1da855a71d14673ef2dc6e00151629ff7ff601a166eb3913288 WatchSource:0}: Error finding container e9771d55e062f1da855a71d14673ef2dc6e00151629ff7ff601a166eb3913288: Status 404 returned error can't find the container with id e9771d55e062f1da855a71d14673ef2dc6e00151629ff7ff601a166eb3913288
Nov 02 07:52:13.083148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:13.083120    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"]
Nov 02 07:52:13.350416 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:13.350324    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn" event={"ID":"4d20daab-f0d4-4f9c-ac1d-70bca6c37618","Type":"ContainerStarted","Data":"e9771d55e062f1da855a71d14673ef2dc6e00151629ff7ff601a166eb3913288"}
Nov 02 07:52:14.361559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:14.361469    4898 generic.go:356] "Generic (PLEG): container finished" podID="4d20daab-f0d4-4f9c-ac1d-70bca6c37618" containerID="46b80982b3a1fad95b1b8d76b3bb1a508347975069d780a892006e4cd15299fc" exitCode=0
Nov 02 07:52:14.362091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:14.361553    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn" event={"ID":"4d20daab-f0d4-4f9c-ac1d-70bca6c37618","Type":"ContainerDied","Data":"46b80982b3a1fad95b1b8d76b3bb1a508347975069d780a892006e4cd15299fc"}
Nov 02 07:52:14.991492 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:52:14.991450    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:52:15.666023 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:15.665993    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:15.743261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:15.743156    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-kube-api-access-cpt4s\" (UniqueName: \"kubernetes.io/projected/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-kube-api-access-cpt4s\") pod \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\" (UID: \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\") "
Nov 02 07:52:15.743426 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:15.743318    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-openshift-ca\") pod \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\" (UID: \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\") "
Nov 02 07:52:15.743426 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:15.743382    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-xks-ca\") pod \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\" (UID: \"4d20daab-f0d4-4f9c-ac1d-70bca6c37618\") "
Nov 02 07:52:15.743927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:15.743883    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-openshift-ca" pod "4d20daab-f0d4-4f9c-ac1d-70bca6c37618" (UID: "4d20daab-f0d4-4f9c-ac1d-70bca6c37618"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:52:15.744140 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:15.744110    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:15.745863 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:15.745834    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-xks-ca" pod "4d20daab-f0d4-4f9c-ac1d-70bca6c37618" (UID: "4d20daab-f0d4-4f9c-ac1d-70bca6c37618"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:52:15.748361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:15.748335    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-kube-api-access-cpt4s" pod "4d20daab-f0d4-4f9c-ac1d-70bca6c37618" (UID: "4d20daab-f0d4-4f9c-ac1d-70bca6c37618"). InnerVolumeSpecName "kube-api-access-cpt4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:15.845193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:15.845160    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpt4s\" (UniqueName: \"kubernetes.io/projected/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-kube-api-access-cpt4s\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:15.845193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:15.845191    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d20daab-f0d4-4f9c-ac1d-70bca6c37618-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:16.444227 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.444191    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"
Nov 02 07:52:16.444227 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.444223    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn" event={"ID":"4d20daab-f0d4-4f9c-ac1d-70bca6c37618","Type":"ContainerDied","Data":"e9771d55e062f1da855a71d14673ef2dc6e00151629ff7ff601a166eb3913288"}
Nov 02 07:52:16.444445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.444268    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9771d55e062f1da855a71d14673ef2dc6e00151629ff7ff601a166eb3913288"
Nov 02 07:52:16.523403 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.523371    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"]
Nov 02 07:52:16.524096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.524074    4898 cpu_manager.go:414] "containerMap: removing container" podUID="4d20daab-f0d4-4f9c-ac1d-70bca6c37618" containerName="revoke-keys"
Nov 02 07:52:16.524096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.524096    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d20daab-f0d4-4f9c-ac1d-70bca6c37618" containerName="revoke-keys"
Nov 02 07:52:16.524290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.524280    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d20daab-f0d4-4f9c-ac1d-70bca6c37618" containerName="revoke-keys"
Nov 02 07:52:16.528209 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.528188    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:16.546901 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.546865    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:52:16.547048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.547032    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:52:16.547321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.547302    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:52:16.547428 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.547389    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:52:16.549567 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.549546    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"]
Nov 02 07:52:16.555492 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.555466    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5145011a-tlsbs\" (UID: \"b7df9796-fb9b-451d-b366-559ecf56a6ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:16.555743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.555722    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxt5n\" (UniqueName: \"kubernetes.io/projected/b7df9796-fb9b-451d-b366-559ecf56a6ab-kube-api-access-dxt5n\") pod \"maas-api-revoke-sub-5145011a-tlsbs\" (UID: \"b7df9796-fb9b-451d-b366-559ecf56a6ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:16.556041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.556018    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5145011a-tlsbs\" (UID: \"b7df9796-fb9b-451d-b366-559ecf56a6ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:16.658427 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.658388    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxt5n\" (UniqueName: \"kubernetes.io/projected/b7df9796-fb9b-451d-b366-559ecf56a6ab-kube-api-access-dxt5n\") pod \"maas-api-revoke-sub-5145011a-tlsbs\" (UID: \"b7df9796-fb9b-451d-b366-559ecf56a6ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:16.658821 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.658802    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5145011a-tlsbs\" (UID: \"b7df9796-fb9b-451d-b366-559ecf56a6ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:16.659105 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.659088    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5145011a-tlsbs\" (UID: \"b7df9796-fb9b-451d-b366-559ecf56a6ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:16.664938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.662085    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5145011a-tlsbs\" (UID: \"b7df9796-fb9b-451d-b366-559ecf56a6ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:16.670949 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.670864    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5145011a-tlsbs\" (UID: \"b7df9796-fb9b-451d-b366-559ecf56a6ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:16.683391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.683298    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxt5n\" (UniqueName: \"kubernetes.io/projected/b7df9796-fb9b-451d-b366-559ecf56a6ab-kube-api-access-dxt5n\") pod \"maas-api-revoke-sub-5145011a-tlsbs\" (UID: \"b7df9796-fb9b-451d-b366-559ecf56a6ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:16.847003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:16.845612    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:17.217491 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:17.217402    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"]
Nov 02 07:52:17.286309 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:17.285276    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"
Nov 02 07:52:17.467717 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:17.467597    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs" event={"ID":"b7df9796-fb9b-451d-b366-559ecf56a6ab","Type":"ContainerStarted","Data":"51419bf5b28adcfe2e405e22ce845600cda6fe8256071023788c0824061d44c7"}
Nov 02 07:52:18.468470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:18.468436    4898 generic.go:356] "Generic (PLEG): container finished" podID="b7df9796-fb9b-451d-b366-559ecf56a6ab" containerID="ecd526c55205128f6540d22ab63fe68c210a6da2f703d62b5d8d8620eb93368c" exitCode=0
Nov 02 07:52:18.469008 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:18.468513    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs" event={"ID":"b7df9796-fb9b-451d-b366-559ecf56a6ab","Type":"ContainerDied","Data":"ecd526c55205128f6540d22ab63fe68c210a6da2f703d62b5d8d8620eb93368c"}
Nov 02 07:52:19.642908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:19.642841    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:19.721134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:19.721097    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7df9796-fb9b-451d-b366-559ecf56a6ab-kube-api-access-dxt5n\" (UniqueName: \"kubernetes.io/projected/b7df9796-fb9b-451d-b366-559ecf56a6ab-kube-api-access-dxt5n\") pod \"b7df9796-fb9b-451d-b366-559ecf56a6ab\" (UID: \"b7df9796-fb9b-451d-b366-559ecf56a6ab\") "
Nov 02 07:52:19.721134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:19.721135    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-xks-ca\") pod \"b7df9796-fb9b-451d-b366-559ecf56a6ab\" (UID: \"b7df9796-fb9b-451d-b366-559ecf56a6ab\") "
Nov 02 07:52:19.721418 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:19.721391    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-openshift-ca\") pod \"b7df9796-fb9b-451d-b366-559ecf56a6ab\" (UID: \"b7df9796-fb9b-451d-b366-559ecf56a6ab\") "
Nov 02 07:52:19.721931 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:19.721846    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-openshift-ca" pod "b7df9796-fb9b-451d-b366-559ecf56a6ab" (UID: "b7df9796-fb9b-451d-b366-559ecf56a6ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:52:19.722079 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:19.722056    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:19.723769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:19.723741    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7df9796-fb9b-451d-b366-559ecf56a6ab-kube-api-access-dxt5n" pod "b7df9796-fb9b-451d-b366-559ecf56a6ab" (UID: "b7df9796-fb9b-451d-b366-559ecf56a6ab"). InnerVolumeSpecName "kube-api-access-dxt5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:19.724098 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:19.724075    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-xks-ca" pod "b7df9796-fb9b-451d-b366-559ecf56a6ab" (UID: "b7df9796-fb9b-451d-b366-559ecf56a6ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:52:19.822864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:19.822836    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxt5n\" (UniqueName: \"kubernetes.io/projected/b7df9796-fb9b-451d-b366-559ecf56a6ab-kube-api-access-dxt5n\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:19.822864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:19.822863    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7df9796-fb9b-451d-b366-559ecf56a6ab-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:20.486665 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.486639    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"
Nov 02 07:52:20.486853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.486673    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs" event={"ID":"b7df9796-fb9b-451d-b366-559ecf56a6ab","Type":"ContainerDied","Data":"51419bf5b28adcfe2e405e22ce845600cda6fe8256071023788c0824061d44c7"}
Nov 02 07:52:20.486853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.486724    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51419bf5b28adcfe2e405e22ce845600cda6fe8256071023788c0824061d44c7"
Nov 02 07:52:20.907987 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.907953    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"]
Nov 02 07:52:20.908608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.908593    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b7df9796-fb9b-451d-b366-559ecf56a6ab" containerName="revoke-keys"
Nov 02 07:52:20.908664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.908609    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7df9796-fb9b-451d-b366-559ecf56a6ab" containerName="revoke-keys"
Nov 02 07:52:20.908731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.908722    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7df9796-fb9b-451d-b366-559ecf56a6ab" containerName="revoke-keys"
Nov 02 07:52:20.912333 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.912313    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:20.927695 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.927667    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:52:20.927854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.927740    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:52:20.932809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.932772    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"]
Nov 02 07:52:20.937609 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.937583    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:52:20.937800 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.937668    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:52:20.939730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.939704    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fsfn\" (UniqueName: \"kubernetes.io/projected/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-kube-api-access-2fsfn\") pod \"maas-api-revoke-sub-9ed387da-4qkch\" (UID: \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:20.940522 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.940409    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed387da-4qkch\" (UID: \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:20.940783 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:20.940753    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed387da-4qkch\" (UID: \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:21.041555 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:21.041506    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fsfn\" (UniqueName: \"kubernetes.io/projected/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-kube-api-access-2fsfn\") pod \"maas-api-revoke-sub-9ed387da-4qkch\" (UID: \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:21.041555 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:21.041565    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed387da-4qkch\" (UID: \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:21.041839 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:21.041635    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed387da-4qkch\" (UID: \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:21.042418 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:21.042392    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed387da-4qkch\" (UID: \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:21.044148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:21.044114    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed387da-4qkch\" (UID: \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:21.064686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:21.064659    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fsfn\" (UniqueName: \"kubernetes.io/projected/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-kube-api-access-2fsfn\") pod \"maas-api-revoke-sub-9ed387da-4qkch\" (UID: \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:21.232245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:21.232166    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:21.539766 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:52:21.539743    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c141c40_6b3e_4c57_9810_0b0f1987c2d8.slice/crio-d400ecf5d671bb13f00f47fc19bc2df4c9534d1033c632214f2fffcf81a7126b WatchSource:0}: Error finding container d400ecf5d671bb13f00f47fc19bc2df4c9534d1033c632214f2fffcf81a7126b: Status 404 returned error can't find the container with id d400ecf5d671bb13f00f47fc19bc2df4c9534d1033c632214f2fffcf81a7126b
Nov 02 07:52:21.541209 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:21.541184    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"]
Nov 02 07:52:22.504625 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:22.504592    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch" event={"ID":"3c141c40-6b3e-4c57-9810-0b0f1987c2d8","Type":"ContainerStarted","Data":"a655d993d1b3bf7f40250bfa9966d845b977c85ecf83e2c32937411fc3fb422a"}
Nov 02 07:52:22.505106 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:22.504644    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch" event={"ID":"3c141c40-6b3e-4c57-9810-0b0f1987c2d8","Type":"ContainerStarted","Data":"d400ecf5d671bb13f00f47fc19bc2df4c9534d1033c632214f2fffcf81a7126b"}
Nov 02 07:52:22.546316 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:22.546254    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch" podStartSLOduration=2.546236005 podStartE2EDuration="2.546236005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:22.544760403 +0000 UTC m=+1693.235087367" watchObservedRunningTime="2026-11-02 07:52:22.546236005 +0000 UTC m=+1693.236562970"
Nov 02 07:52:23.517155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:23.517116    4898 generic.go:356] "Generic (PLEG): container finished" podID="3c141c40-6b3e-4c57-9810-0b0f1987c2d8" containerID="a655d993d1b3bf7f40250bfa9966d845b977c85ecf83e2c32937411fc3fb422a" exitCode=0
Nov 02 07:52:23.517632 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:23.517166    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch" event={"ID":"3c141c40-6b3e-4c57-9810-0b0f1987c2d8","Type":"ContainerDied","Data":"a655d993d1b3bf7f40250bfa9966d845b977c85ecf83e2c32937411fc3fb422a"}
Nov 02 07:52:24.728833 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:24.728804    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:24.793635 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:24.793566    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-xks-ca\") pod \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\" (UID: \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\") "
Nov 02 07:52:24.793997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:24.793699    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-kube-api-access-2fsfn\" (UniqueName: \"kubernetes.io/projected/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-kube-api-access-2fsfn\") pod \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\" (UID: \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\") "
Nov 02 07:52:24.793997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:24.793736    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-openshift-ca\") pod \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\" (UID: \"3c141c40-6b3e-4c57-9810-0b0f1987c2d8\") "
Nov 02 07:52:24.804846 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:24.804713    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-openshift-ca" pod "3c141c40-6b3e-4c57-9810-0b0f1987c2d8" (UID: "3c141c40-6b3e-4c57-9810-0b0f1987c2d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:52:24.805636 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:24.805588    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-kube-api-access-2fsfn" pod "3c141c40-6b3e-4c57-9810-0b0f1987c2d8" (UID: "3c141c40-6b3e-4c57-9810-0b0f1987c2d8"). InnerVolumeSpecName "kube-api-access-2fsfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:24.808229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:24.808204    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-xks-ca" pod "3c141c40-6b3e-4c57-9810-0b0f1987c2d8" (UID: "3c141c40-6b3e-4c57-9810-0b0f1987c2d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:52:24.895786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:24.895751    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fsfn\" (UniqueName: \"kubernetes.io/projected/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-kube-api-access-2fsfn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:24.895786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:24.895781    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:24.896020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:24.895816    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c141c40-6b3e-4c57-9810-0b0f1987c2d8-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:25.347055 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.346992    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"]
Nov 02 07:52:25.349649 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.348253    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3c141c40-6b3e-4c57-9810-0b0f1987c2d8" containerName="revoke-keys"
Nov 02 07:52:25.349649 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.348278    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c141c40-6b3e-4c57-9810-0b0f1987c2d8" containerName="revoke-keys"
Nov 02 07:52:25.351077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.348461    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c141c40-6b3e-4c57-9810-0b0f1987c2d8" containerName="revoke-keys"
Nov 02 07:52:25.361594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.361570    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:25.366458 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.366429    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"]
Nov 02 07:52:25.405159 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.405128    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a9e5c3-9lz4s\" (UID: \"38878f90-ab7b-4eb3-ad7b-0f600288e498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:25.405477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.405453    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a9e5c3-9lz4s\" (UID: \"38878f90-ab7b-4eb3-ad7b-0f600288e498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:25.405669 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.405644    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvqww\" (UniqueName: \"kubernetes.io/projected/38878f90-ab7b-4eb3-ad7b-0f600288e498-kube-api-access-rvqww\") pod \"maas-api-revoke-sub-67a9e5c3-9lz4s\" (UID: \"38878f90-ab7b-4eb3-ad7b-0f600288e498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:25.506774 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.506715    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvqww\" (UniqueName: \"kubernetes.io/projected/38878f90-ab7b-4eb3-ad7b-0f600288e498-kube-api-access-rvqww\") pod \"maas-api-revoke-sub-67a9e5c3-9lz4s\" (UID: \"38878f90-ab7b-4eb3-ad7b-0f600288e498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:25.506774 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.506787    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a9e5c3-9lz4s\" (UID: \"38878f90-ab7b-4eb3-ad7b-0f600288e498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:25.507054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.506947    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a9e5c3-9lz4s\" (UID: \"38878f90-ab7b-4eb3-ad7b-0f600288e498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:25.507705 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.507673    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a9e5c3-9lz4s\" (UID: \"38878f90-ab7b-4eb3-ad7b-0f600288e498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:25.509803 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.509775    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a9e5c3-9lz4s\" (UID: \"38878f90-ab7b-4eb3-ad7b-0f600288e498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:25.530799 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.530771    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvqww\" (UniqueName: \"kubernetes.io/projected/38878f90-ab7b-4eb3-ad7b-0f600288e498-kube-api-access-rvqww\") pod \"maas-api-revoke-sub-67a9e5c3-9lz4s\" (UID: \"38878f90-ab7b-4eb3-ad7b-0f600288e498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:25.555994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.555970    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"
Nov 02 07:52:25.556145 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.555973    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch" event={"ID":"3c141c40-6b3e-4c57-9810-0b0f1987c2d8","Type":"ContainerDied","Data":"d400ecf5d671bb13f00f47fc19bc2df4c9534d1033c632214f2fffcf81a7126b"}
Nov 02 07:52:25.556145 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.556043    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d400ecf5d671bb13f00f47fc19bc2df4c9534d1033c632214f2fffcf81a7126b"
Nov 02 07:52:25.686517 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.686491    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:25.989079 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:52:25.989045    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38878f90_ab7b_4eb3_ad7b_0f600288e498.slice/crio-c839ccff9a07454570d37d0e56e7c623a7b43c0e1abf1c4c986265f0fe19860a WatchSource:0}: Error finding container c839ccff9a07454570d37d0e56e7c623a7b43c0e1abf1c4c986265f0fe19860a: Status 404 returned error can't find the container with id c839ccff9a07454570d37d0e56e7c623a7b43c0e1abf1c4c986265f0fe19860a
Nov 02 07:52:25.994263 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:25.994236    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"]
Nov 02 07:52:26.567325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:26.567228    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s" event={"ID":"38878f90-ab7b-4eb3-ad7b-0f600288e498","Type":"ContainerStarted","Data":"ec8b385745bead599f0c98172d2114d38059e9704071846cc76f12f896a5f787"}
Nov 02 07:52:26.567325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:26.567299    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s" event={"ID":"38878f90-ab7b-4eb3-ad7b-0f600288e498","Type":"ContainerStarted","Data":"c839ccff9a07454570d37d0e56e7c623a7b43c0e1abf1c4c986265f0fe19860a"}
Nov 02 07:52:26.611823 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:26.611768    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s" podStartSLOduration=1.611753526 podStartE2EDuration="1.611753526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:26.607542256 +0000 UTC m=+1697.297869224" watchObservedRunningTime="2026-11-02 07:52:26.611753526 +0000 UTC m=+1697.302080618"
Nov 02 07:52:27.577539 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:27.577497    4898 generic.go:356] "Generic (PLEG): container finished" podID="38878f90-ab7b-4eb3-ad7b-0f600288e498" containerID="ec8b385745bead599f0c98172d2114d38059e9704071846cc76f12f896a5f787" exitCode=0
Nov 02 07:52:27.578044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:27.577640    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s" event={"ID":"38878f90-ab7b-4eb3-ad7b-0f600288e498","Type":"ContainerDied","Data":"ec8b385745bead599f0c98172d2114d38059e9704071846cc76f12f896a5f787"}
Nov 02 07:52:28.751760 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:28.751733    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:28.854700 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:28.854622    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-xks-ca\") pod \"38878f90-ab7b-4eb3-ad7b-0f600288e498\" (UID: \"38878f90-ab7b-4eb3-ad7b-0f600288e498\") "
Nov 02 07:52:28.854878 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:28.854702    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38878f90-ab7b-4eb3-ad7b-0f600288e498-kube-api-access-rvqww\" (UniqueName: \"kubernetes.io/projected/38878f90-ab7b-4eb3-ad7b-0f600288e498-kube-api-access-rvqww\") pod \"38878f90-ab7b-4eb3-ad7b-0f600288e498\" (UID: \"38878f90-ab7b-4eb3-ad7b-0f600288e498\") "
Nov 02 07:52:28.855012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:28.854977    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-openshift-ca\") pod \"38878f90-ab7b-4eb3-ad7b-0f600288e498\" (UID: \"38878f90-ab7b-4eb3-ad7b-0f600288e498\") "
Nov 02 07:52:28.855332 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:28.855302    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-openshift-ca" pod "38878f90-ab7b-4eb3-ad7b-0f600288e498" (UID: "38878f90-ab7b-4eb3-ad7b-0f600288e498"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:52:28.855514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:28.855496    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:28.857426 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:28.857399    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-xks-ca" pod "38878f90-ab7b-4eb3-ad7b-0f600288e498" (UID: "38878f90-ab7b-4eb3-ad7b-0f600288e498"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:52:28.857518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:28.857426    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38878f90-ab7b-4eb3-ad7b-0f600288e498-kube-api-access-rvqww" pod "38878f90-ab7b-4eb3-ad7b-0f600288e498" (UID: "38878f90-ab7b-4eb3-ad7b-0f600288e498"). InnerVolumeSpecName "kube-api-access-rvqww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:28.962770 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:28.957043    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38878f90-ab7b-4eb3-ad7b-0f600288e498-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:28.962770 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:28.957091    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvqww\" (UniqueName: \"kubernetes.io/projected/38878f90-ab7b-4eb3-ad7b-0f600288e498-kube-api-access-rvqww\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:29.597493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:29.597461    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"
Nov 02 07:52:29.597689 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:29.597463    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s" event={"ID":"38878f90-ab7b-4eb3-ad7b-0f600288e498","Type":"ContainerDied","Data":"c839ccff9a07454570d37d0e56e7c623a7b43c0e1abf1c4c986265f0fe19860a"}
Nov 02 07:52:29.597689 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:29.597603    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c839ccff9a07454570d37d0e56e7c623a7b43c0e1abf1c4c986265f0fe19860a"
Nov 02 07:52:34.594913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:34.583984    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:34.651912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:34.650964    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.124913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:35.123964    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.534915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:35.534454    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.588914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:35.588137    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.588914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:35.588688    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:52:35.622168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:35.622126    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.660034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:35.656974    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl" podUID="147b04eb-2c14-4316-914e-788b949ccb50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:35.660034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:35.657430    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:52:35.685039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:35.685001    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:52:36.108914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.095501    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"]
Nov 02 07:52:36.108914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.096485    4898 cpu_manager.go:414] "containerMap: removing container" podUID="38878f90-ab7b-4eb3-ad7b-0f600288e498" containerName="revoke-keys"
Nov 02 07:52:36.108914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.096507    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38878f90-ab7b-4eb3-ad7b-0f600288e498" containerName="revoke-keys"
Nov 02 07:52:36.108914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.096700    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="38878f90-ab7b-4eb3-ad7b-0f600288e498" containerName="revoke-keys"
Nov 02 07:52:36.146916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.121197    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:36.199927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.125139    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded"
Nov 02 07:52:36.199927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.126676    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfrdb\" (UniqueName: \"kubernetes.io/projected/ce951366-9ab8-4843-8b17-e9ddb906df18-kube-api-access-lfrdb\") pod \"maas-api-revoke-sub-bc5365d8-hrgc6\" (UID: \"ce951366-9ab8-4843-8b17-e9ddb906df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:36.199927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.184661    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc5365d8-hrgc6\" (UID: \"ce951366-9ab8-4843-8b17-e9ddb906df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:36.199927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.184823    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc5365d8-hrgc6\" (UID: \"ce951366-9ab8-4843-8b17-e9ddb906df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:36.199927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.130653    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"]
Nov 02 07:52:36.199927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.185096    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:52:36.258137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.206739    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:52:36.258137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.237006    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:52:36.258137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.237115    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:52:36.258137 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.237256    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl"
Nov 02 07:52:36.285937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.285827    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc5365d8-hrgc6\" (UID: \"ce951366-9ab8-4843-8b17-e9ddb906df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:36.287874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.286348    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfrdb\" (UniqueName: \"kubernetes.io/projected/ce951366-9ab8-4843-8b17-e9ddb906df18-kube-api-access-lfrdb\") pod \"maas-api-revoke-sub-bc5365d8-hrgc6\" (UID: \"ce951366-9ab8-4843-8b17-e9ddb906df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:36.287874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.286433    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc5365d8-hrgc6\" (UID: \"ce951366-9ab8-4843-8b17-e9ddb906df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:36.287874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.287337    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc5365d8-hrgc6\" (UID: \"ce951366-9ab8-4843-8b17-e9ddb906df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:36.287874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.287772    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:52:36.382877 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.382760    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfrdb\" (UniqueName: \"kubernetes.io/projected/ce951366-9ab8-4843-8b17-e9ddb906df18-kube-api-access-lfrdb\") pod \"maas-api-revoke-sub-bc5365d8-hrgc6\" (UID: \"ce951366-9ab8-4843-8b17-e9ddb906df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:36.483831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.467306    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc5365d8-hrgc6\" (UID: \"ce951366-9ab8-4843-8b17-e9ddb906df18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:36.486049 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.485970    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 07:52:36.511290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.511242    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 07:52:36.570123 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.569844    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:36.910070 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:52:36.909989    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce951366_9ab8_4843_8b17_e9ddb906df18.slice/crio-8b2676e934887c7e789e99b412801498bc01993c8642219930cfe7c5ca411bf1 WatchSource:0}: Error finding container 8b2676e934887c7e789e99b412801498bc01993c8642219930cfe7c5ca411bf1: Status 404 returned error can't find the container with id 8b2676e934887c7e789e99b412801498bc01993c8642219930cfe7c5ca411bf1
Nov 02 07:52:36.913352 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:36.913241    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"]
Nov 02 07:52:37.585756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.583564    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"]
Nov 02 07:52:37.623591 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.622440    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"]
Nov 02 07:52:37.623591 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.622611    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:37.733429 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.733394    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60a36ede-55fhv\" (UID: \"c81edb37-6116-4ca9-aa4f-06283464d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:37.733612 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.733592    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60a36ede-55fhv\" (UID: \"c81edb37-6116-4ca9-aa4f-06283464d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:37.733697 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.733684    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6g6h\" (UniqueName: \"kubernetes.io/projected/c81edb37-6116-4ca9-aa4f-06283464d900-kube-api-access-k6g6h\") pod \"maas-api-revoke-sub-60a36ede-55fhv\" (UID: \"c81edb37-6116-4ca9-aa4f-06283464d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:37.834526 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.834496    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60a36ede-55fhv\" (UID: \"c81edb37-6116-4ca9-aa4f-06283464d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:37.834683 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.834565    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6g6h\" (UniqueName: \"kubernetes.io/projected/c81edb37-6116-4ca9-aa4f-06283464d900-kube-api-access-k6g6h\") pod \"maas-api-revoke-sub-60a36ede-55fhv\" (UID: \"c81edb37-6116-4ca9-aa4f-06283464d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:37.834683 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.834637    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60a36ede-55fhv\" (UID: \"c81edb37-6116-4ca9-aa4f-06283464d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:37.835169 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.835144    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60a36ede-55fhv\" (UID: \"c81edb37-6116-4ca9-aa4f-06283464d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:37.836967 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.836882    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60a36ede-55fhv\" (UID: \"c81edb37-6116-4ca9-aa4f-06283464d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:37.862455 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.862427    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6g6h\" (UniqueName: \"kubernetes.io/projected/c81edb37-6116-4ca9-aa4f-06283464d900-kube-api-access-k6g6h\") pod \"maas-api-revoke-sub-60a36ede-55fhv\" (UID: \"c81edb37-6116-4ca9-aa4f-06283464d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:37.869981 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.869952    4898 generic.go:356] "Generic (PLEG): container finished" podID="ce951366-9ab8-4843-8b17-e9ddb906df18" containerID="0495c2368cfdd09ceae43d16a9d4425dedb0dc77c965872d3f2851da9d919590" exitCode=0
Nov 02 07:52:37.870114 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.870012    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6" event={"ID":"ce951366-9ab8-4843-8b17-e9ddb906df18","Type":"ContainerDied","Data":"0495c2368cfdd09ceae43d16a9d4425dedb0dc77c965872d3f2851da9d919590"}
Nov 02 07:52:37.870114 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.870069    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6" event={"ID":"ce951366-9ab8-4843-8b17-e9ddb906df18","Type":"ContainerStarted","Data":"8b2676e934887c7e789e99b412801498bc01993c8642219930cfe7c5ca411bf1"}
Nov 02 07:52:37.942431 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:37.942393    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:38.252268 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:52:38.252062    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81edb37_6116_4ca9_aa4f_06283464d900.slice/crio-faaa494a1b0642a905e7f2a9be3b48fcad104e6ccbafa1c5b0dab43c983c64cf WatchSource:0}: Error finding container faaa494a1b0642a905e7f2a9be3b48fcad104e6ccbafa1c5b0dab43c983c64cf: Status 404 returned error can't find the container with id faaa494a1b0642a905e7f2a9be3b48fcad104e6ccbafa1c5b0dab43c983c64cf
Nov 02 07:52:38.253650 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:38.253615    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"]
Nov 02 07:52:38.882030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:38.881922    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv" event={"ID":"c81edb37-6116-4ca9-aa4f-06283464d900","Type":"ContainerStarted","Data":"e691c6d92d383a65221dc7134357631daa7ed5a1c7824b7de6ea2182df95ef50"}
Nov 02 07:52:38.882030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:38.881993    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv" event={"ID":"c81edb37-6116-4ca9-aa4f-06283464d900","Type":"ContainerStarted","Data":"faaa494a1b0642a905e7f2a9be3b48fcad104e6ccbafa1c5b0dab43c983c64cf"}
Nov 02 07:52:38.927077 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:38.927019    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv" podStartSLOduration=1.927004417 podStartE2EDuration="1.927004417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:38.924087329 +0000 UTC m=+1709.614414295" watchObservedRunningTime="2026-11-02 07:52:38.927004417 +0000 UTC m=+1709.617331380"
Nov 02 07:52:39.034060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.034032    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:39.170962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.166414    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-xks-ca\") pod \"ce951366-9ab8-4843-8b17-e9ddb906df18\" (UID: \"ce951366-9ab8-4843-8b17-e9ddb906df18\") "
Nov 02 07:52:39.170962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.166515    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce951366-9ab8-4843-8b17-e9ddb906df18-kube-api-access-lfrdb\" (UniqueName: \"kubernetes.io/projected/ce951366-9ab8-4843-8b17-e9ddb906df18-kube-api-access-lfrdb\") pod \"ce951366-9ab8-4843-8b17-e9ddb906df18\" (UID: \"ce951366-9ab8-4843-8b17-e9ddb906df18\") "
Nov 02 07:52:39.170962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.166826    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-openshift-ca\") pod \"ce951366-9ab8-4843-8b17-e9ddb906df18\" (UID: \"ce951366-9ab8-4843-8b17-e9ddb906df18\") "
Nov 02 07:52:39.170962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.167936    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-openshift-ca" pod "ce951366-9ab8-4843-8b17-e9ddb906df18" (UID: "ce951366-9ab8-4843-8b17-e9ddb906df18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:52:39.181204 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.181169    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce951366-9ab8-4843-8b17-e9ddb906df18-kube-api-access-lfrdb" pod "ce951366-9ab8-4843-8b17-e9ddb906df18" (UID: "ce951366-9ab8-4843-8b17-e9ddb906df18"). InnerVolumeSpecName "kube-api-access-lfrdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:39.201353 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.201315    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-xks-ca" pod "ce951366-9ab8-4843-8b17-e9ddb906df18" (UID: "ce951366-9ab8-4843-8b17-e9ddb906df18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:52:39.268571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.268537    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:39.268571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.268565    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce951366-9ab8-4843-8b17-e9ddb906df18-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:39.268571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.268575    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfrdb\" (UniqueName: \"kubernetes.io/projected/ce951366-9ab8-4843-8b17-e9ddb906df18-kube-api-access-lfrdb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:39.891956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.891806    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"
Nov 02 07:52:39.891956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.891820    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6" event={"ID":"ce951366-9ab8-4843-8b17-e9ddb906df18","Type":"ContainerDied","Data":"8b2676e934887c7e789e99b412801498bc01993c8642219930cfe7c5ca411bf1"}
Nov 02 07:52:39.891956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.891882    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b2676e934887c7e789e99b412801498bc01993c8642219930cfe7c5ca411bf1"
Nov 02 07:52:39.898387 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.898347    4898 generic.go:356] "Generic (PLEG): container finished" podID="c81edb37-6116-4ca9-aa4f-06283464d900" containerID="e691c6d92d383a65221dc7134357631daa7ed5a1c7824b7de6ea2182df95ef50" exitCode=0
Nov 02 07:52:39.898651 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:39.898623    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv" event={"ID":"c81edb37-6116-4ca9-aa4f-06283464d900","Type":"ContainerDied","Data":"e691c6d92d383a65221dc7134357631daa7ed5a1c7824b7de6ea2182df95ef50"}
Nov 02 07:52:41.298013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.297979    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:41.399787 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.399750    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81edb37-6116-4ca9-aa4f-06283464d900-kube-api-access-k6g6h\" (UniqueName: \"kubernetes.io/projected/c81edb37-6116-4ca9-aa4f-06283464d900-kube-api-access-k6g6h\") pod \"c81edb37-6116-4ca9-aa4f-06283464d900\" (UID: \"c81edb37-6116-4ca9-aa4f-06283464d900\") "
Nov 02 07:52:41.400009 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.399991    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-openshift-ca\") pod \"c81edb37-6116-4ca9-aa4f-06283464d900\" (UID: \"c81edb37-6116-4ca9-aa4f-06283464d900\") "
Nov 02 07:52:41.400083 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.400027    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-xks-ca\") pod \"c81edb37-6116-4ca9-aa4f-06283464d900\" (UID: \"c81edb37-6116-4ca9-aa4f-06283464d900\") "
Nov 02 07:52:41.400734 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.400702    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-openshift-ca" pod "c81edb37-6116-4ca9-aa4f-06283464d900" (UID: "c81edb37-6116-4ca9-aa4f-06283464d900"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:52:41.402218 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.402191    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-xks-ca" pod "c81edb37-6116-4ca9-aa4f-06283464d900" (UID: "c81edb37-6116-4ca9-aa4f-06283464d900"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:52:41.402686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.402659    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81edb37-6116-4ca9-aa4f-06283464d900-kube-api-access-k6g6h" pod "c81edb37-6116-4ca9-aa4f-06283464d900" (UID: "c81edb37-6116-4ca9-aa4f-06283464d900"). InnerVolumeSpecName "kube-api-access-k6g6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:52:41.501812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.501777    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:41.501812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.501811    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81edb37-6116-4ca9-aa4f-06283464d900-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:41.502142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.501826    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6g6h\" (UniqueName: \"kubernetes.io/projected/c81edb37-6116-4ca9-aa4f-06283464d900-kube-api-access-k6g6h\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:52:41.848954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.848923    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64_3db07204-2fc3-4be1-9730-5f47f311a5ae/payload-processing/0.log"
Nov 02 07:52:41.973583 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.973550    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv" event={"ID":"c81edb37-6116-4ca9-aa4f-06283464d900","Type":"ContainerDied","Data":"faaa494a1b0642a905e7f2a9be3b48fcad104e6ccbafa1c5b0dab43c983c64cf"}
Nov 02 07:52:41.973583 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.973577    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"
Nov 02 07:52:41.973784 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:41.973601    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faaa494a1b0642a905e7f2a9be3b48fcad104e6ccbafa1c5b0dab43c983c64cf"
Nov 02 07:52:42.033880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:42.033846    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64_3db07204-2fc3-4be1-9730-5f47f311a5ae/payload-processing/0.log"
Nov 02 07:52:44.208013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.207975    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"]
Nov 02 07:52:44.208954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.208912    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c81edb37-6116-4ca9-aa4f-06283464d900" containerName="revoke-keys"
Nov 02 07:52:44.208954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.208953    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81edb37-6116-4ca9-aa4f-06283464d900" containerName="revoke-keys"
Nov 02 07:52:44.209151 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.208996    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ce951366-9ab8-4843-8b17-e9ddb906df18" containerName="revoke-keys"
Nov 02 07:52:44.209151 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.209006    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce951366-9ab8-4843-8b17-e9ddb906df18" containerName="revoke-keys"
Nov 02 07:52:44.209285 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.209199    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81edb37-6116-4ca9-aa4f-06283464d900" containerName="revoke-keys"
Nov 02 07:52:44.209285 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.209218    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce951366-9ab8-4843-8b17-e9ddb906df18" containerName="revoke-keys"
Nov 02 07:52:44.213138 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.213117    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.237200 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.237170    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"]
Nov 02 07:52:44.240790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.240649    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.240790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.240704    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233e4b67-1d34-446a-b44a-6ee2f2731df5-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.240790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.240738    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.240790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.240776    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.241208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.240904    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.241208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.240954    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.241208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.240999    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.241208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.241094    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.241208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.241129    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qczgc\" (UniqueName: \"kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-kube-api-access-qczgc\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.242827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.242798    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9ab4d2-openshift-default-dockercfg-xtcp6\""
Nov 02 07:52:44.342290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.342253    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.342472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.342301    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.342472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.342329    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.342472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.342389    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.342472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.342419    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczgc\" (UniqueName: \"kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-kube-api-access-qczgc\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.342707 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.342512    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.342707 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.342646    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233e4b67-1d34-446a-b44a-6ee2f2731df5-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.342842 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.342716    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.342842 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.342697    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.342842 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.342778    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.342842 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.342789    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.343069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.342850    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.343123 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.343102    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.343299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.343278    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233e4b67-1d34-446a-b44a-6ee2f2731df5-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.344965 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.344938    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.345111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.345092    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.366133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.366111    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.366233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.366156    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczgc\" (UniqueName: \"kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-kube-api-access-qczgc\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.529237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.529141    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:44.851251 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:52:44.851167    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233e4b67_1d34_446a_b44a_6ee2f2731df5.slice/crio-1f5f9e3695bf259dae907664696a8b5b5c19537b98375c68a689fd4c251861f2 WatchSource:0}: Error finding container 1f5f9e3695bf259dae907664696a8b5b5c19537b98375c68a689fd4c251861f2: Status 404 returned error can't find the container with id 1f5f9e3695bf259dae907664696a8b5b5c19537b98375c68a689fd4c251861f2
Nov 02 07:52:44.854210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.854174    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:44.854327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.854260    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:44.854327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.854309    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:44.855654 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:44.855630    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"]
Nov 02 07:52:45.013909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:45.013022    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796" event={"ID":"233e4b67-1d34-446a-b44a-6ee2f2731df5","Type":"ContainerStarted","Data":"402c54bed947226c4a24289c87000297d9610137a9bf3e2bf5309b574ffe792d"}
Nov 02 07:52:45.013909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:45.013096    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796" event={"ID":"233e4b67-1d34-446a-b44a-6ee2f2731df5","Type":"ContainerStarted","Data":"1f5f9e3695bf259dae907664696a8b5b5c19537b98375c68a689fd4c251861f2"}
Nov 02 07:52:45.029665 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:52:45.029633    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:52:45.061242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:45.061189    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796" podStartSLOduration=1.061176584 podStartE2EDuration="1.061176584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:45.05884135 +0000 UTC m=+1715.749168313" watchObservedRunningTime="2026-11-02 07:52:45.061176584 +0000 UTC m=+1715.751503547"
Nov 02 07:52:45.529841 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:45.529807    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:45.531374 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:45.531340    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 07:52:45.531517 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:45.531399    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796" podUID="233e4b67-1d34-446a-b44a-6ee2f2731df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 07:52:46.529679 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:46.529546    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 07:52:46.529679 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:46.529609    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796" podUID="233e4b67-1d34-446a-b44a-6ee2f2731df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 07:52:47.529531 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:47.529492    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 07:52:47.529939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:47.529554    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796" podUID="233e4b67-1d34-446a-b44a-6ee2f2731df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 07:52:48.533018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:48.532980    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 07:52:48.533619 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:48.533585    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796" podUID="233e4b67-1d34-446a-b44a-6ee2f2731df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 07:52:49.449735 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.449701    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"]
Nov 02 07:52:49.504268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.503129    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.530003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.526823    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-certs\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.530003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.527095    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lws9\" (UniqueName: \"kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-kube-api-access-5lws9\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.530003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.527136    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-data\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.530003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.527184    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-credential-socket\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.530003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.527249    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-socket\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.530003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.527293    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-podinfo\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.530003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.527322    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-envoy\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.530003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.527391    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-token\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.530003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.527485    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d446180-273b-4b72-9dcf-e0979ddbe681-istiod-ca-cert\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.530003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.528024    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-9877ce-openshift-default-dockercfg-9gv72\""
Nov 02 07:52:49.535933 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.531100    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 07:52:49.535933 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.531160    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796" podUID="233e4b67-1d34-446a-b44a-6ee2f2731df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 07:52:49.536671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.536650    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"]
Nov 02 07:52:49.641912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.631710    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-token\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.641912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.631776    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d446180-273b-4b72-9dcf-e0979ddbe681-istiod-ca-cert\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.641912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.631954    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-certs\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.641912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.632067    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lws9\" (UniqueName: \"kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-kube-api-access-5lws9\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.641912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.632096    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-data\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.641912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.632136    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-credential-socket\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.641912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.632195    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-socket\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.641912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.632263    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-podinfo\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.641912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.632291    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-envoy\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.658915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.650543    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d446180-273b-4b72-9dcf-e0979ddbe681-istiod-ca-cert\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.658915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.651110    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-certs\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.658915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.651479    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-socket\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.658915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.651687    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-data\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.658915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.652017    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-credential-socket\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.681092 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.677278    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-envoy\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.699955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.695071    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-podinfo\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.732916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.732344    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lws9\" (UniqueName: \"kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-kube-api-access-5lws9\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.746909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.734029    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-token\") pod \"e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") " pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:49.829915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:49.829157    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:50.531549 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:50.531134    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 07:52:50.531549 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:50.531191    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796" podUID="233e4b67-1d34-446a-b44a-6ee2f2731df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 07:52:50.719065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:50.719024    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:50.719584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:50.719560    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:50.719702 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:50.719686    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:52:50.730810 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:50.729822    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"]
Nov 02 07:52:51.264376 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:51.260476    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" event={"ID":"0d446180-273b-4b72-9dcf-e0979ddbe681","Type":"ContainerStarted","Data":"4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a"}
Nov 02 07:52:51.264376 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:51.260591    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" event={"ID":"0d446180-273b-4b72-9dcf-e0979ddbe681","Type":"ContainerStarted","Data":"007d77fd20a7fe55561107b2fe094a0abf6f0f91a0d000a5baa6e090e0942e28"}
Nov 02 07:52:51.347005 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:51.344397    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" podStartSLOduration=2.344369449 podStartE2EDuration="2.344369449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:51.328946838 +0000 UTC m=+1722.019273805" watchObservedRunningTime="2026-11-02 07:52:51.344369449 +0000 UTC m=+1722.034696414"
Nov 02 07:52:51.591245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:51.591162    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:51.592564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:51.592539    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:51.594041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:51.594007    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:52:51.830602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:51.830561    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:51.837214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:51.837182    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:52.271124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:52.271091    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:52.272153 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:52.272133    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:52:55.636904 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:55.636860    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"]
Nov 02 07:52:55.641943 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:55.641916    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"
Nov 02 07:52:55.666999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:55.666964    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"]
Nov 02 07:52:55.673733 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:55.673698    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-9877ce-dockercfg-78kdx\""
Nov 02 07:52:55.674557 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:55.674533    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-9877ce\""
Nov 02 07:52:55.702287 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:55.702246    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbr7p\" (UniqueName: \"kubernetes.io/projected/0f07eefc-2028-4748-823e-8b0ec22ac61e-kube-api-access-dbr7p\") pod \"payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl\" (UID: \"0f07eefc-2028-4748-823e-8b0ec22ac61e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"
Nov 02 07:52:55.702559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:55.702529    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f07eefc-2028-4748-823e-8b0ec22ac61e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl\" (UID: \"0f07eefc-2028-4748-823e-8b0ec22ac61e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"
Nov 02 07:52:55.806113 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:55.806070    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f07eefc-2028-4748-823e-8b0ec22ac61e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl\" (UID: \"0f07eefc-2028-4748-823e-8b0ec22ac61e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"
Nov 02 07:52:55.821347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:55.821300    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbr7p\" (UniqueName: \"kubernetes.io/projected/0f07eefc-2028-4748-823e-8b0ec22ac61e-kube-api-access-dbr7p\") pod \"payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl\" (UID: \"0f07eefc-2028-4748-823e-8b0ec22ac61e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"
Nov 02 07:52:55.838027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:55.809071    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f07eefc-2028-4748-823e-8b0ec22ac61e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl\" (UID: \"0f07eefc-2028-4748-823e-8b0ec22ac61e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"
Nov 02 07:52:55.875916 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:55.874108    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbr7p\" (UniqueName: \"kubernetes.io/projected/0f07eefc-2028-4748-823e-8b0ec22ac61e-kube-api-access-dbr7p\") pod \"payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl\" (UID: \"0f07eefc-2028-4748-823e-8b0ec22ac61e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"
Nov 02 07:52:55.961790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:55.961751    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"
Nov 02 07:52:56.266588 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:52:56.266560    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f07eefc_2028_4748_823e_8b0ec22ac61e.slice/crio-822a204aed6cf9b6367910a5eced737aa4146fb30c2a48b4c68d1899ae566861 WatchSource:0}: Error finding container 822a204aed6cf9b6367910a5eced737aa4146fb30c2a48b4c68d1899ae566861: Status 404 returned error can't find the container with id 822a204aed6cf9b6367910a5eced737aa4146fb30c2a48b4c68d1899ae566861
Nov 02 07:52:56.270797 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:56.270769    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"]
Nov 02 07:52:56.310256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:56.310230    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl" event={"ID":"0f07eefc-2028-4748-823e-8b0ec22ac61e","Type":"ContainerStarted","Data":"822a204aed6cf9b6367910a5eced737aa4146fb30c2a48b4c68d1899ae566861"}
Nov 02 07:52:57.324361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:57.324320    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl" event={"ID":"0f07eefc-2028-4748-823e-8b0ec22ac61e","Type":"ContainerStarted","Data":"72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee"}
Nov 02 07:52:57.325007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:57.324980    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"
Nov 02 07:52:57.368336 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:57.368267    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl" podStartSLOduration=2.3682484 podStartE2EDuration="2.3682484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:52:57.366734113 +0000 UTC m=+1728.057061077" watchObservedRunningTime="2026-11-02 07:52:57.3682484 +0000 UTC m=+1728.058575366"
Nov 02 07:52:58.480518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:52:58.480455    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"]
Nov 02 07:53:00.738101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:00.738056    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"]
Nov 02 07:53:00.798105 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:00.798065    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"]
Nov 02 07:53:03.481693 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:03.481645    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk" podUID="060dc90e-d6de-4546-9af1-c0a809e3e8a6" containerName="payload-pre-processing" containerID="cri-o://bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d" gracePeriod=25
Nov 02 07:53:03.781147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:03.781113    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"
Nov 02 07:53:03.811518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:03.811490    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060dc90e-d6de-4546-9af1-c0a809e3e8a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060dc90e-d6de-4546-9af1-c0a809e3e8a6-plugins-config-volume\") pod \"060dc90e-d6de-4546-9af1-c0a809e3e8a6\" (UID: \"060dc90e-d6de-4546-9af1-c0a809e3e8a6\") "
Nov 02 07:53:03.811664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:03.811539    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060dc90e-d6de-4546-9af1-c0a809e3e8a6-kube-api-access-tl4nl\" (UniqueName: \"kubernetes.io/projected/060dc90e-d6de-4546-9af1-c0a809e3e8a6-kube-api-access-tl4nl\") pod \"060dc90e-d6de-4546-9af1-c0a809e3e8a6\" (UID: \"060dc90e-d6de-4546-9af1-c0a809e3e8a6\") "
Nov 02 07:53:03.811924 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:03.811874    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060dc90e-d6de-4546-9af1-c0a809e3e8a6-plugins-config-volume" pod "060dc90e-d6de-4546-9af1-c0a809e3e8a6" (UID: "060dc90e-d6de-4546-9af1-c0a809e3e8a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:03.812053 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:03.812005    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060dc90e-d6de-4546-9af1-c0a809e3e8a6-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:03.813671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:03.813649    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060dc90e-d6de-4546-9af1-c0a809e3e8a6-kube-api-access-tl4nl" pod "060dc90e-d6de-4546-9af1-c0a809e3e8a6" (UID: "060dc90e-d6de-4546-9af1-c0a809e3e8a6"). InnerVolumeSpecName "kube-api-access-tl4nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:03.913647 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:03.913616    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl4nl\" (UniqueName: \"kubernetes.io/projected/060dc90e-d6de-4546-9af1-c0a809e3e8a6-kube-api-access-tl4nl\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:04.476248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:04.476212    4898 generic.go:356] "Generic (PLEG): container finished" podID="060dc90e-d6de-4546-9af1-c0a809e3e8a6" containerID="bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d" exitCode=0
Nov 02 07:53:04.476436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:04.476273    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"
Nov 02 07:53:04.476436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:04.476304    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk" event={"ID":"060dc90e-d6de-4546-9af1-c0a809e3e8a6","Type":"ContainerDied","Data":"bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d"}
Nov 02 07:53:04.476436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:04.476358    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk" event={"ID":"060dc90e-d6de-4546-9af1-c0a809e3e8a6","Type":"ContainerDied","Data":"99ed71905d058d81b522a6ffc63e34fe40b41230430ffd15a063cbf447ac9719"}
Nov 02 07:53:04.476436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:04.476382    4898 scope.go:122] "RemoveContainer" containerID="bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d"
Nov 02 07:53:04.489905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:04.489689    4898 scope.go:122] "RemoveContainer" containerID="bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d"
Nov 02 07:53:04.490207 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:53:04.490042    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d\": container with ID starting with bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d not found: ID does not exist" containerID="bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d"
Nov 02 07:53:04.490207 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:04.490072    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d"} err="failed to get container status \"bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d\": rpc error: code = NotFound desc = could not find container \"bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d\": container with ID starting with bffab9d7207130e38e18feb67abf3046c970dfcfcf218378937042752b1d039d not found: ID does not exist"
Nov 02 07:53:04.535039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:04.535012    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"]
Nov 02 07:53:04.544382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:04.544353    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-11b2e0-fc9fb8dcf-s68lk"]
Nov 02 07:53:05.468902 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:05.468857    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060dc90e-d6de-4546-9af1-c0a809e3e8a6" path="/var/lib/kubelet/pods/060dc90e-d6de-4546-9af1-c0a809e3e8a6/volumes"
Nov 02 07:53:05.739844 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:05.739755    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm" podUID="f2dd73d7-b5ce-4ab6-916b-02dd18cbf177" containerName="payload-processing" containerID="cri-o://2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f" gracePeriod=25
Nov 02 07:53:05.798682 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:05.798611    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g" podUID="95858c5b-0b54-4563-9b6e-05e241a67e10" containerName="payload-pre-processing" containerID="cri-o://f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0" gracePeriod=25
Nov 02 07:53:06.101116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.101084    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"
Nov 02 07:53:06.143275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.143247    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-kube-api-access-d7gtl\" (UniqueName: \"kubernetes.io/projected/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-kube-api-access-d7gtl\") pod \"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177\" (UID: \"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177\") "
Nov 02 07:53:06.144119 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.143519    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-plugins-config-volume\") pod \"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177\" (UID: \"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177\") "
Nov 02 07:53:06.144119 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.143929    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-plugins-config-volume" pod "f2dd73d7-b5ce-4ab6-916b-02dd18cbf177" (UID: "f2dd73d7-b5ce-4ab6-916b-02dd18cbf177"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:06.144119 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.144091    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:06.145605 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.145578    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-kube-api-access-d7gtl" pod "f2dd73d7-b5ce-4ab6-916b-02dd18cbf177" (UID: "f2dd73d7-b5ce-4ab6-916b-02dd18cbf177"). InnerVolumeSpecName "kube-api-access-d7gtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:06.149096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.149073    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"
Nov 02 07:53:06.245090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.245051    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95858c5b-0b54-4563-9b6e-05e241a67e10-kube-api-access-csmvc\" (UniqueName: \"kubernetes.io/projected/95858c5b-0b54-4563-9b6e-05e241a67e10-kube-api-access-csmvc\") pod \"95858c5b-0b54-4563-9b6e-05e241a67e10\" (UID: \"95858c5b-0b54-4563-9b6e-05e241a67e10\") "
Nov 02 07:53:06.245301 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.245114    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95858c5b-0b54-4563-9b6e-05e241a67e10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95858c5b-0b54-4563-9b6e-05e241a67e10-plugins-config-volume\") pod \"95858c5b-0b54-4563-9b6e-05e241a67e10\" (UID: \"95858c5b-0b54-4563-9b6e-05e241a67e10\") "
Nov 02 07:53:06.245614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.245581    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95858c5b-0b54-4563-9b6e-05e241a67e10-plugins-config-volume" pod "95858c5b-0b54-4563-9b6e-05e241a67e10" (UID: "95858c5b-0b54-4563-9b6e-05e241a67e10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:06.245747 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.245690    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95858c5b-0b54-4563-9b6e-05e241a67e10-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:06.245747 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.245714    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7gtl\" (UniqueName: \"kubernetes.io/projected/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177-kube-api-access-d7gtl\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:06.247943 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.247904    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95858c5b-0b54-4563-9b6e-05e241a67e10-kube-api-access-csmvc" pod "95858c5b-0b54-4563-9b6e-05e241a67e10" (UID: "95858c5b-0b54-4563-9b6e-05e241a67e10"). InnerVolumeSpecName "kube-api-access-csmvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:06.346923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.346832    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csmvc\" (UniqueName: \"kubernetes.io/projected/95858c5b-0b54-4563-9b6e-05e241a67e10-kube-api-access-csmvc\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:06.499810 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.499770    4898 generic.go:356] "Generic (PLEG): container finished" podID="f2dd73d7-b5ce-4ab6-916b-02dd18cbf177" containerID="2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f" exitCode=0
Nov 02 07:53:06.500032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.499812    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"
Nov 02 07:53:06.500032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.499847    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm" event={"ID":"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177","Type":"ContainerDied","Data":"2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f"}
Nov 02 07:53:06.500032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.499942    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm" event={"ID":"f2dd73d7-b5ce-4ab6-916b-02dd18cbf177","Type":"ContainerDied","Data":"79247ad723beaeb43cbe90e288356594e00adb49bb1201d4fe8a36da5e5ad4a9"}
Nov 02 07:53:06.500032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.499977    4898 scope.go:122] "RemoveContainer" containerID="2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f"
Nov 02 07:53:06.501996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.501783    4898 generic.go:356] "Generic (PLEG): container finished" podID="95858c5b-0b54-4563-9b6e-05e241a67e10" containerID="f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0" exitCode=0
Nov 02 07:53:06.501996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.501839    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"
Nov 02 07:53:06.501996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.501917    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g" event={"ID":"95858c5b-0b54-4563-9b6e-05e241a67e10","Type":"ContainerDied","Data":"f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0"}
Nov 02 07:53:06.501996 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.501962    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g" event={"ID":"95858c5b-0b54-4563-9b6e-05e241a67e10","Type":"ContainerDied","Data":"6eca17f21ccdd0e63fb7b0dec18e37ae966ea9fbaf7f9f1a762bc635932a87cd"}
Nov 02 07:53:06.517156 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.517136    4898 scope.go:122] "RemoveContainer" containerID="2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f"
Nov 02 07:53:06.521132 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:53:06.521104    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f\": container with ID starting with 2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f not found: ID does not exist" containerID="2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f"
Nov 02 07:53:06.521247 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.521143    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f"} err="failed to get container status \"2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f\": rpc error: code = NotFound desc = could not find container \"2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f\": container with ID starting with 2dedbc292e25525c27a097dafb36cbab3cd813e5f2c921fa4b52ceacdc44365f not found: ID does not exist"
Nov 02 07:53:06.521247 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.521161    4898 scope.go:122] "RemoveContainer" containerID="f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0"
Nov 02 07:53:06.531535 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.531514    4898 scope.go:122] "RemoveContainer" containerID="f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0"
Nov 02 07:53:06.531804 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:53:06.531787    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0\": container with ID starting with f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0 not found: ID does not exist" containerID="f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0"
Nov 02 07:53:06.531872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.531812    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0"} err="failed to get container status \"f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0\": rpc error: code = NotFound desc = could not find container \"f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0\": container with ID starting with f21a417274c7eb4803ecfa9f594c26714d8e00d1d2ea92df4408b38e58a14ab0 not found: ID does not exist"
Nov 02 07:53:06.554870 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.554835    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"]
Nov 02 07:53:06.588907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.584733    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3624d8-7d7f54d85c-k6fpm"]
Nov 02 07:53:06.630047 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.629952    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"]
Nov 02 07:53:06.635209 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:06.635182    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3624d8-694f97cbcf-hl76g"]
Nov 02 07:53:07.467519 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:07.467489    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95858c5b-0b54-4563-9b6e-05e241a67e10" path="/var/lib/kubelet/pods/95858c5b-0b54-4563-9b6e-05e241a67e10/volumes"
Nov 02 07:53:07.468022 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:07.467922    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2dd73d7-b5ce-4ab6-916b-02dd18cbf177" path="/var/lib/kubelet/pods/f2dd73d7-b5ce-4ab6-916b-02dd18cbf177/volumes"
Nov 02 07:53:08.390325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:08.390299    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"
Nov 02 07:53:09.484552 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:09.484522    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"]
Nov 02 07:53:09.485056 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:09.484762    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy" containerID="cri-o://e9b07d639b6bf3f4ab753cc1550bc066016e85787909831fdbd881296682667e" gracePeriod=30
Nov 02 07:53:11.328595 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:11.328567    4898 scope.go:122] "RemoveContainer" containerID="1003b2e05258003e59509a784df5ba85f90ebd1abde144d7c0a88db0ae171e2e"
Nov 02 07:53:14.293256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.293216    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"]
Nov 02 07:53:14.294250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.294226    4898 cpu_manager.go:414] "containerMap: removing container" podUID="060dc90e-d6de-4546-9af1-c0a809e3e8a6" containerName="payload-pre-processing"
Nov 02 07:53:14.294250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.294249    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060dc90e-d6de-4546-9af1-c0a809e3e8a6" containerName="payload-pre-processing"
Nov 02 07:53:14.294424 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.294267    4898 cpu_manager.go:414] "containerMap: removing container" podUID="95858c5b-0b54-4563-9b6e-05e241a67e10" containerName="payload-pre-processing"
Nov 02 07:53:14.294424 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.294274    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95858c5b-0b54-4563-9b6e-05e241a67e10" containerName="payload-pre-processing"
Nov 02 07:53:14.294424 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.294340    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f2dd73d7-b5ce-4ab6-916b-02dd18cbf177" containerName="payload-processing"
Nov 02 07:53:14.294424 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.294348    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dd73d7-b5ce-4ab6-916b-02dd18cbf177" containerName="payload-processing"
Nov 02 07:53:14.294633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.294530    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dd73d7-b5ce-4ab6-916b-02dd18cbf177" containerName="payload-processing"
Nov 02 07:53:14.294633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.294548    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="060dc90e-d6de-4546-9af1-c0a809e3e8a6" containerName="payload-pre-processing"
Nov 02 07:53:14.294633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.294559    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="95858c5b-0b54-4563-9b6e-05e241a67e10" containerName="payload-pre-processing"
Nov 02 07:53:14.300388 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.300361    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.338000 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.337966    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"]
Nov 02 07:53:14.346881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.346852    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"]
Nov 02 07:53:14.352769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.352743    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.363410 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.363374    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"]
Nov 02 07:53:14.377610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.377576    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.377800 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.377682    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.377800 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.377783    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4gqn\" (UniqueName: \"kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-kube-api-access-v4gqn\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.377953 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.377820    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b4be44-0446-4e93-b48d-63e17726be91-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.378021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.377948    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.378021 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.377979    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.378133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.378109    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.378186 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.378171    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.378261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.378246    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b4be44-0446-4e93-b48d-63e17726be91-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.479636 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.479602    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.479831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.479651    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.479919 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.479825    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.479992 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.479925    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b4be44-0446-4e93-b48d-63e17726be91-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.479992 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.479972    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkgk6\" (UniqueName: \"kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-kube-api-access-mkgk6\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.480102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480020    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.480193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480176    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.480254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480209    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.480254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480216    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.480379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480307    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.480379 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480334    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.480489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480411    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.480489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480458    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.480489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480480    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4gqn\" (UniqueName: \"kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-kube-api-access-v4gqn\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.480641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480533    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.480641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480582    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b4be44-0446-4e93-b48d-63e17726be91-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.480641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480618    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.480771 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480672    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.480771 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480747    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.480872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480809    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.480872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.480858    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.481078 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.481062    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.481415 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.481388    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b4be44-0446-4e93-b48d-63e17726be91-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.482668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.482640    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b4be44-0446-4e93-b48d-63e17726be91-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.483134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.483112    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.503288 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.503265    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.504318 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.504297    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4gqn\" (UniqueName: \"kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-kube-api-access-v4gqn\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.582293 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.582184    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.582293 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.582279    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.582541 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.582357    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.582541 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.582433    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.582541 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.582488    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.582703 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.582576    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.584102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.583162    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-data\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.584102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.583192    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.584102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.583261    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkgk6\" (UniqueName: \"kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-kube-api-access-mkgk6\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.584102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.583312    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.584102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.583171    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-credential-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.584102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.583943    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istiod-ca-cert\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.584102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.584060    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-socket\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.584102 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.584101    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-certs\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.585718 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.585696    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-podinfo\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.586905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.586870    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-envoy\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.606835 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.606746    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkgk6\" (UniqueName: \"kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-kube-api-access-mkgk6\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.607355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.607331    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-token\") pod \"e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") " pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.625950 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.616951    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:14.658217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.650798    4898 generic.go:356] "Generic (PLEG): container finished" podID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerID="e9b07d639b6bf3f4ab753cc1550bc066016e85787909831fdbd881296682667e" exitCode=0
Nov 02 07:53:14.658217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.651127    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" event={"ID":"76acab59-e07d-49ae-9075-0accdbe6e5b1","Type":"ContainerDied","Data":"e9b07d639b6bf3f4ab753cc1550bc066016e85787909831fdbd881296682667e"}
Nov 02 07:53:14.658217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.651191    4898 scope.go:122] "RemoveContainer" containerID="663ae3b37b0b77c21928ec2af6d43b955fd884efb9f0c5cbef4654b460da4987"
Nov 02 07:53:14.677821 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.677789    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:14.810067 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.810035    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:53:14.887091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887013    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-kube-api-access-wgkfz\" (UniqueName: \"kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-kube-api-access-wgkfz\") pod \"76acab59-e07d-49ae-9075-0accdbe6e5b1\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") "
Nov 02 07:53:14.887234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887121    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-data\") pod \"76acab59-e07d-49ae-9075-0accdbe6e5b1\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") "
Nov 02 07:53:14.887234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887169    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-socket\") pod \"76acab59-e07d-49ae-9075-0accdbe6e5b1\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") "
Nov 02 07:53:14.887234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887212    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-credential-socket\") pod \"76acab59-e07d-49ae-9075-0accdbe6e5b1\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") "
Nov 02 07:53:14.887412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887262    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-envoy\") pod \"76acab59-e07d-49ae-9075-0accdbe6e5b1\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") "
Nov 02 07:53:14.887412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887309    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-token\" (UniqueName: \"kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-token\") pod \"76acab59-e07d-49ae-9075-0accdbe6e5b1\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") "
Nov 02 07:53:14.887412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887335    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76acab59-e07d-49ae-9075-0accdbe6e5b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76acab59-e07d-49ae-9075-0accdbe6e5b1-istiod-ca-cert\") pod \"76acab59-e07d-49ae-9075-0accdbe6e5b1\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") "
Nov 02 07:53:14.887412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887380    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-podinfo\") pod \"76acab59-e07d-49ae-9075-0accdbe6e5b1\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") "
Nov 02 07:53:14.887642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887480    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-data" pod "76acab59-e07d-49ae-9075-0accdbe6e5b1" (UID: "76acab59-e07d-49ae-9075-0accdbe6e5b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:14.887642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887496    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-certs\") pod \"76acab59-e07d-49ae-9075-0accdbe6e5b1\" (UID: \"76acab59-e07d-49ae-9075-0accdbe6e5b1\") "
Nov 02 07:53:14.887755 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887723    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-credential-socket" pod "76acab59-e07d-49ae-9075-0accdbe6e5b1" (UID: "76acab59-e07d-49ae-9075-0accdbe6e5b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:14.887821 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887801    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76acab59-e07d-49ae-9075-0accdbe6e5b1-istiod-ca-cert" pod "76acab59-e07d-49ae-9075-0accdbe6e5b1" (UID: "76acab59-e07d-49ae-9075-0accdbe6e5b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:14.888017 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.887982    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-socket" pod "76acab59-e07d-49ae-9075-0accdbe6e5b1" (UID: "76acab59-e07d-49ae-9075-0accdbe6e5b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:14.888017 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.888010    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-certs" pod "76acab59-e07d-49ae-9075-0accdbe6e5b1" (UID: "76acab59-e07d-49ae-9075-0accdbe6e5b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:14.888359 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.888336    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:14.888445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.888365    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:14.888445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.888379    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76acab59-e07d-49ae-9075-0accdbe6e5b1-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:14.888445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.888393    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:14.888445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.888406    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:14.890171 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.890149    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-token" pod "76acab59-e07d-49ae-9075-0accdbe6e5b1" (UID: "76acab59-e07d-49ae-9075-0accdbe6e5b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:14.890382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.890365    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-envoy" pod "76acab59-e07d-49ae-9075-0accdbe6e5b1" (UID: "76acab59-e07d-49ae-9075-0accdbe6e5b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:14.890562 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.890538    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-kube-api-access-wgkfz" pod "76acab59-e07d-49ae-9075-0accdbe6e5b1" (UID: "76acab59-e07d-49ae-9075-0accdbe6e5b1"). InnerVolumeSpecName "kube-api-access-wgkfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:14.890630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.890542    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-podinfo" pod "76acab59-e07d-49ae-9075-0accdbe6e5b1" (UID: "76acab59-e07d-49ae-9075-0accdbe6e5b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:53:14.973039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.973012    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"]
Nov 02 07:53:14.973599 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.973559    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:14.973651 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.973641    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:14.973709 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.973697    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:14.990520 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.990488    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgkfz\" (UniqueName: \"kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-kube-api-access-wgkfz\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:14.990520 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.990522    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:14.990739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.990535    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:14.990739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:14.990552    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76acab59-e07d-49ae-9075-0accdbe6e5b1-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:14.999827 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:53:14.999778    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:53:15.016514 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:53:15.016485    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b4d4f7_1bc1_42e2_8add_e43fc90e8b4c.slice/crio-b98eb3740960a37b47c837a47fa47eb4b0c0a8c4c3f7fb8f11be3d5afed92073 WatchSource:0}: Error finding container b98eb3740960a37b47c837a47fa47eb4b0c0a8c4c3f7fb8f11be3d5afed92073: Status 404 returned error can't find the container with id b98eb3740960a37b47c837a47fa47eb4b0c0a8c4c3f7fb8f11be3d5afed92073
Nov 02 07:53:15.020938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.020850    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:15.021070 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.020998    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:15.021070 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.021053    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:15.021725 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.021701    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"]
Nov 02 07:53:15.723025 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.722980    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw" event={"ID":"76acab59-e07d-49ae-9075-0accdbe6e5b1","Type":"ContainerDied","Data":"d560d131b143ee93dfea944f1f2dafc464ced442db072d4fc8bce2befc5206a2"}
Nov 02 07:53:15.723510 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.723106    4898 scope.go:122] "RemoveContainer" containerID="e9b07d639b6bf3f4ab753cc1550bc066016e85787909831fdbd881296682667e"
Nov 02 07:53:15.723510 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.723341    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"
Nov 02 07:53:15.730731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.730697    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd" event={"ID":"d2b4be44-0446-4e93-b48d-63e17726be91","Type":"ContainerStarted","Data":"a52f02d75cb9226dfa6a67f86fc1cf58657707685cdadce1f3f881fe07041f4c"}
Nov 02 07:53:15.730977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.730956    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd" event={"ID":"d2b4be44-0446-4e93-b48d-63e17726be91","Type":"ContainerStarted","Data":"45297959d0dbf5647923781ced759f3d53993bcafb90808f8706a7b08e407af4"}
Nov 02 07:53:15.741442 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.741418    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77" event={"ID":"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c","Type":"ContainerStarted","Data":"3444669bd923d6547eb769e4191ee5a9ae011b22767ce00c5bb21ebae4a20fdd"}
Nov 02 07:53:15.742035 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.742006    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77" event={"ID":"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c","Type":"ContainerStarted","Data":"b98eb3740960a37b47c837a47fa47eb4b0c0a8c4c3f7fb8f11be3d5afed92073"}
Nov 02 07:53:15.821966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.817927    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"]
Nov 02 07:53:15.830028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.829988    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3624d8-openshift-default-6556748985-6f9hw"]
Nov 02 07:53:15.847609 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.847561    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd" podStartSLOduration=1.8475476020000001 podStartE2EDuration="1.847547602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:15.843852233 +0000 UTC m=+1746.534179197" watchObservedRunningTime="2026-11-02 07:53:15.847547602 +0000 UTC m=+1746.537874560"
Nov 02 07:53:15.928321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:15.928261    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77" podStartSLOduration=1.92824557 podStartE2EDuration="1.92824557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:15.924088883 +0000 UTC m=+1746.614415851" watchObservedRunningTime="2026-11-02 07:53:15.92824557 +0000 UTC m=+1746.618572534"
Nov 02 07:53:16.201928 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:16.200934    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"]
Nov 02 07:53:16.635383 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:16.634053    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:16.644329 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:16.642792    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 07:53:16.644329 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:16.642863    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd" podUID="d2b4be44-0446-4e93-b48d-63e17726be91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 07:53:16.687911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:16.686881    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:16.711915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:16.694992    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 07:53:16.711915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:16.695062    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77" podUID="27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 07:53:17.527283 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:17.526198    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" path="/var/lib/kubelet/pods/76acab59-e07d-49ae-9075-0accdbe6e5b1/volumes"
Nov 02 07:53:17.627449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:17.627377    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 07:53:17.627623 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:17.627523    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd" podUID="d2b4be44-0446-4e93-b48d-63e17726be91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 07:53:17.680911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:17.680341    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 07:53:17.680911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:17.680420    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77" podUID="27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 07:53:18.621914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:18.621231    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 07:53:18.621914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:18.621295    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd" podUID="d2b4be44-0446-4e93-b48d-63e17726be91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 07:53:18.700027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:18.699991    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 07:53:18.700325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:18.700295    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77" podUID="27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 07:53:19.620219 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:19.620181    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 07:53:19.620661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:19.620630    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd" podUID="d2b4be44-0446-4e93-b48d-63e17726be91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 07:53:19.684925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:19.684272    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 07:53:19.684925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:19.684330    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77" podUID="27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 07:53:20.709602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:20.656179    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"]
Nov 02 07:53:20.709602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:20.656469    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerName="istio-proxy" containerID="cri-o://7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc" gracePeriod=30
Nov 02 07:53:20.709602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:20.671783    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 07:53:20.709602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:20.671854    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd" podUID="d2b4be44-0446-4e93-b48d-63e17726be91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 07:53:20.709602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:20.682867    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 07:53:20.709602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:20.682931    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77" podUID="27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 07:53:21.209033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.208956    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq" podUID="d429cbf5-924c-4ba9-91dd-c0b877835f24" containerName="payload-pre-processing" containerID="cri-o://e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458" gracePeriod=25
Nov 02 07:53:21.637752 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.637718    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"
Nov 02 07:53:21.755134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.754353    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d429cbf5-924c-4ba9-91dd-c0b877835f24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d429cbf5-924c-4ba9-91dd-c0b877835f24-plugins-config-volume\") pod \"d429cbf5-924c-4ba9-91dd-c0b877835f24\" (UID: \"d429cbf5-924c-4ba9-91dd-c0b877835f24\") "
Nov 02 07:53:21.755134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.754423    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d429cbf5-924c-4ba9-91dd-c0b877835f24-plugins-config-volume" pod "d429cbf5-924c-4ba9-91dd-c0b877835f24" (UID: "d429cbf5-924c-4ba9-91dd-c0b877835f24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:21.755134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.754498    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d429cbf5-924c-4ba9-91dd-c0b877835f24-kube-api-access-7jv8m\" (UniqueName: \"kubernetes.io/projected/d429cbf5-924c-4ba9-91dd-c0b877835f24-kube-api-access-7jv8m\") pod \"d429cbf5-924c-4ba9-91dd-c0b877835f24\" (UID: \"d429cbf5-924c-4ba9-91dd-c0b877835f24\") "
Nov 02 07:53:21.755134 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.755053    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d429cbf5-924c-4ba9-91dd-c0b877835f24-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:21.758031 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.757984    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d429cbf5-924c-4ba9-91dd-c0b877835f24-kube-api-access-7jv8m" pod "d429cbf5-924c-4ba9-91dd-c0b877835f24" (UID: "d429cbf5-924c-4ba9-91dd-c0b877835f24"). InnerVolumeSpecName "kube-api-access-7jv8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:21.870629 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.870486    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jv8m\" (UniqueName: \"kubernetes.io/projected/d429cbf5-924c-4ba9-91dd-c0b877835f24-kube-api-access-7jv8m\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:21.915290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.913287    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:21.915290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.915144    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:21.916269 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.916243    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:21.967257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.967227    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:21.967731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.967705    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:21.971737 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:21.971715    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:22.128453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:22.128357    4898 generic.go:356] "Generic (PLEG): container finished" podID="d429cbf5-924c-4ba9-91dd-c0b877835f24" containerID="e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458" exitCode=0
Nov 02 07:53:22.128453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:22.128406    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"
Nov 02 07:53:22.128453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:22.128439    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq" event={"ID":"d429cbf5-924c-4ba9-91dd-c0b877835f24","Type":"ContainerDied","Data":"e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458"}
Nov 02 07:53:22.128714 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:22.128498    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq" event={"ID":"d429cbf5-924c-4ba9-91dd-c0b877835f24","Type":"ContainerDied","Data":"3987bc7e37f2ea4ec9f9fbdcaba2bc460b8675d652e71f574dc3ee4f43d57a09"}
Nov 02 07:53:22.128714 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:22.128526    4898 scope.go:122] "RemoveContainer" containerID="e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458"
Nov 02 07:53:22.140305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:22.140282    4898 scope.go:122] "RemoveContainer" containerID="e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458"
Nov 02 07:53:22.140629 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:53:22.140606    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458\": container with ID starting with e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458 not found: ID does not exist" containerID="e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458"
Nov 02 07:53:22.140742 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:22.140642    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458"} err="failed to get container status \"e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458\": rpc error: code = NotFound desc = could not find container \"e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458\": container with ID starting with e9cd75e14376ee4510a1cb428f423d53884647bd05a1f8abe572f4e88eacd458 not found: ID does not exist"
Nov 02 07:53:22.186816 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:22.186787    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"]
Nov 02 07:53:22.192955 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:22.192930    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9ab4d2-6856cc9f8c-lrdvq"]
Nov 02 07:53:23.466492 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:23.466459    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d429cbf5-924c-4ba9-91dd-c0b877835f24" path="/var/lib/kubelet/pods/d429cbf5-924c-4ba9-91dd-c0b877835f24/volumes"
Nov 02 07:53:25.081905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.081857    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"]
Nov 02 07:53:25.082693 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.082669    4898 cpu_manager.go:414] "containerMap: removing container" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy"
Nov 02 07:53:25.082810 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.082694    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy"
Nov 02 07:53:25.082810 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.082735    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d429cbf5-924c-4ba9-91dd-c0b877835f24" containerName="payload-pre-processing"
Nov 02 07:53:25.082810 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.082743    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d429cbf5-924c-4ba9-91dd-c0b877835f24" containerName="payload-pre-processing"
Nov 02 07:53:25.083007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.082971    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy"
Nov 02 07:53:25.083007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.082990    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy"
Nov 02 07:53:25.083108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.083008    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d429cbf5-924c-4ba9-91dd-c0b877835f24" containerName="payload-pre-processing"
Nov 02 07:53:25.087069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.087048    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:25.106929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.106871    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"]
Nov 02 07:53:25.111288 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.111252    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:53:25.111587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.111563    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:53:25.111711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.111590    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:53:25.111711 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.111661    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:53:25.121763 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.121737    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a6cf606-mjwrv\" (UID: \"7d625389-08c3-4009-beb8-68cc7832967f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:25.121900 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.121870    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcxfh\" (UniqueName: \"kubernetes.io/projected/7d625389-08c3-4009-beb8-68cc7832967f-kube-api-access-pcxfh\") pod \"maas-api-revoke-sub-5a6cf606-mjwrv\" (UID: \"7d625389-08c3-4009-beb8-68cc7832967f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:25.122148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.122069    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a6cf606-mjwrv\" (UID: \"7d625389-08c3-4009-beb8-68cc7832967f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:25.223993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.223957    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a6cf606-mjwrv\" (UID: \"7d625389-08c3-4009-beb8-68cc7832967f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:25.224189 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.224087    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a6cf606-mjwrv\" (UID: \"7d625389-08c3-4009-beb8-68cc7832967f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:25.224189 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.224139    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcxfh\" (UniqueName: \"kubernetes.io/projected/7d625389-08c3-4009-beb8-68cc7832967f-kube-api-access-pcxfh\") pod \"maas-api-revoke-sub-5a6cf606-mjwrv\" (UID: \"7d625389-08c3-4009-beb8-68cc7832967f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:25.224657 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.224634    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a6cf606-mjwrv\" (UID: \"7d625389-08c3-4009-beb8-68cc7832967f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:25.226461 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.226443    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a6cf606-mjwrv\" (UID: \"7d625389-08c3-4009-beb8-68cc7832967f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:25.248140 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.248107    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcxfh\" (UniqueName: \"kubernetes.io/projected/7d625389-08c3-4009-beb8-68cc7832967f-kube-api-access-pcxfh\") pod \"maas-api-revoke-sub-5a6cf606-mjwrv\" (UID: \"7d625389-08c3-4009-beb8-68cc7832967f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:25.408371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.408340    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:25.739339 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:53:25.739294    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d625389_08c3_4009_beb8_68cc7832967f.slice/crio-24b2485ac90d953791bedca9e49e19a902ea33f3655750ea3f652da85560f9fb WatchSource:0}: Error finding container 24b2485ac90d953791bedca9e49e19a902ea33f3655750ea3f652da85560f9fb: Status 404 returned error can't find the container with id 24b2485ac90d953791bedca9e49e19a902ea33f3655750ea3f652da85560f9fb
Nov 02 07:53:25.743583 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:25.743556    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"]
Nov 02 07:53:26.207915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.204359    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv" event={"ID":"7d625389-08c3-4009-beb8-68cc7832967f","Type":"ContainerStarted","Data":"c6fe8ec035b9c9fa85c012cbe761c7b7984d5b50530f07078c1713f9eee8b6bd"}
Nov 02 07:53:26.207915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.204446    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv" event={"ID":"7d625389-08c3-4009-beb8-68cc7832967f","Type":"ContainerStarted","Data":"24b2485ac90d953791bedca9e49e19a902ea33f3655750ea3f652da85560f9fb"}
Nov 02 07:53:26.274915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.270250    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv" podStartSLOduration=1.270229808 podStartE2EDuration="1.270229808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:26.261351197 +0000 UTC m=+1756.951678174" watchObservedRunningTime="2026-11-02 07:53:26.270229808 +0000 UTC m=+1756.960556773"
Nov 02 07:53:26.552715 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.552688    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:53:26.802668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.802641    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-token\" (UniqueName: \"kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-token\") pod \"f8a9a6c9-7420-428e-a684-8dba457ecae9\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") "
Nov 02 07:53:26.802823 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.802676    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-credential-socket\") pod \"f8a9a6c9-7420-428e-a684-8dba457ecae9\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") "
Nov 02 07:53:26.802823 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.802698    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-socket\") pod \"f8a9a6c9-7420-428e-a684-8dba457ecae9\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") "
Nov 02 07:53:26.802823 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.802757    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-kube-api-access-9tnlt\" (UniqueName: \"kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-kube-api-access-9tnlt\") pod \"f8a9a6c9-7420-428e-a684-8dba457ecae9\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") "
Nov 02 07:53:26.803018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.802883    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-data\") pod \"f8a9a6c9-7420-428e-a684-8dba457ecae9\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") "
Nov 02 07:53:26.803018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.802952    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a9a6c9-7420-428e-a684-8dba457ecae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a9a6c9-7420-428e-a684-8dba457ecae9-istiod-ca-cert\") pod \"f8a9a6c9-7420-428e-a684-8dba457ecae9\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") "
Nov 02 07:53:26.803018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.802969    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-credential-socket" pod "f8a9a6c9-7420-428e-a684-8dba457ecae9" (UID: "f8a9a6c9-7420-428e-a684-8dba457ecae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:26.803018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.803004    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-envoy\") pod \"f8a9a6c9-7420-428e-a684-8dba457ecae9\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") "
Nov 02 07:53:26.803227 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.803082    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-certs\") pod \"f8a9a6c9-7420-428e-a684-8dba457ecae9\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") "
Nov 02 07:53:26.803227 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.803176    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-podinfo\") pod \"f8a9a6c9-7420-428e-a684-8dba457ecae9\" (UID: \"f8a9a6c9-7420-428e-a684-8dba457ecae9\") "
Nov 02 07:53:26.803608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.803148    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-socket" pod "f8a9a6c9-7420-428e-a684-8dba457ecae9" (UID: "f8a9a6c9-7420-428e-a684-8dba457ecae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:26.803608 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.803210    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-data" pod "f8a9a6c9-7420-428e-a684-8dba457ecae9" (UID: "f8a9a6c9-7420-428e-a684-8dba457ecae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:26.803786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.803438    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-certs" pod "f8a9a6c9-7420-428e-a684-8dba457ecae9" (UID: "f8a9a6c9-7420-428e-a684-8dba457ecae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:26.803786 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.803512    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a9a6c9-7420-428e-a684-8dba457ecae9-istiod-ca-cert" pod "f8a9a6c9-7420-428e-a684-8dba457ecae9" (UID: "f8a9a6c9-7420-428e-a684-8dba457ecae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:26.803956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.803937    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.804042 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.803962    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a9a6c9-7420-428e-a684-8dba457ecae9-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.804042 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.803976    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.804042 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.803988    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.804042 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.804004    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.805416 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.805391    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-envoy" pod "f8a9a6c9-7420-428e-a684-8dba457ecae9" (UID: "f8a9a6c9-7420-428e-a684-8dba457ecae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:26.805516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.805469    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-token" pod "f8a9a6c9-7420-428e-a684-8dba457ecae9" (UID: "f8a9a6c9-7420-428e-a684-8dba457ecae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:26.805661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.805640    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-kube-api-access-9tnlt" pod "f8a9a6c9-7420-428e-a684-8dba457ecae9" (UID: "f8a9a6c9-7420-428e-a684-8dba457ecae9"). InnerVolumeSpecName "kube-api-access-9tnlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:26.805833 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.805817    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-podinfo" pod "f8a9a6c9-7420-428e-a684-8dba457ecae9" (UID: "f8a9a6c9-7420-428e-a684-8dba457ecae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:53:26.905730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.905687    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.905730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.905717    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tnlt\" (UniqueName: \"kubernetes.io/projected/f8a9a6c9-7420-428e-a684-8dba457ecae9-kube-api-access-9tnlt\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.905730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.905727    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:26.905730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:26.905737    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a9a6c9-7420-428e-a684-8dba457ecae9-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:27.218474 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.218442    4898 generic.go:356] "Generic (PLEG): container finished" podID="7d625389-08c3-4009-beb8-68cc7832967f" containerID="c6fe8ec035b9c9fa85c012cbe761c7b7984d5b50530f07078c1713f9eee8b6bd" exitCode=0
Nov 02 07:53:27.219010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.218532    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv" event={"ID":"7d625389-08c3-4009-beb8-68cc7832967f","Type":"ContainerDied","Data":"c6fe8ec035b9c9fa85c012cbe761c7b7984d5b50530f07078c1713f9eee8b6bd"}
Nov 02 07:53:27.220641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.220614    4898 generic.go:356] "Generic (PLEG): container finished" podID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerID="7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc" exitCode=0
Nov 02 07:53:27.220743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.220672    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"
Nov 02 07:53:27.220743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.220691    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" event={"ID":"f8a9a6c9-7420-428e-a684-8dba457ecae9","Type":"ContainerDied","Data":"7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc"}
Nov 02 07:53:27.220743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.220734    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4" event={"ID":"f8a9a6c9-7420-428e-a684-8dba457ecae9","Type":"ContainerDied","Data":"4260a6c8f949bf2e6f66bcac62477979e729c8f5dac1141dd8d8483349663b9d"}
Nov 02 07:53:27.220862 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.220757    4898 scope.go:122] "RemoveContainer" containerID="7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc"
Nov 02 07:53:27.236835 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.236814    4898 scope.go:122] "RemoveContainer" containerID="ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb"
Nov 02 07:53:27.253504 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.253484    4898 scope.go:122] "RemoveContainer" containerID="7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc"
Nov 02 07:53:27.253756 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:53:27.253733    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc\": container with ID starting with 7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc not found: ID does not exist" containerID="7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc"
Nov 02 07:53:27.253872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.253761    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc"} err="failed to get container status \"7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc\": rpc error: code = NotFound desc = could not find container \"7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc\": container with ID starting with 7c3a82a11ecaf0f523f588a2d88df4d1f6697da2189ad24d07091c2db6d471fc not found: ID does not exist"
Nov 02 07:53:27.253872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.253779    4898 scope.go:122] "RemoveContainer" containerID="ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb"
Nov 02 07:53:27.254794 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:53:27.254774    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb\": container with ID starting with ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb not found: ID does not exist" containerID="ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb"
Nov 02 07:53:27.254843 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.254801    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb"} err="failed to get container status \"ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb\": rpc error: code = NotFound desc = could not find container \"ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb\": container with ID starting with ff79449dabc9f5323ff70d325c8e8b06e6e8334bfbd6b2febbd3877317da1ffb not found: ID does not exist"
Nov 02 07:53:27.300026 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.300001    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"]
Nov 02 07:53:27.313563 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.313540    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-11b2e0-openshift-default-69868c7689-tpds4"]
Nov 02 07:53:27.466872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:27.466838    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" path="/var/lib/kubelet/pods/f8a9a6c9-7420-428e-a684-8dba457ecae9/volumes"
Nov 02 07:53:28.516527 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.516488    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"]
Nov 02 07:53:28.517444 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.517424    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerName="istio-proxy"
Nov 02 07:53:28.517546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.517445    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerName="istio-proxy"
Nov 02 07:53:28.517546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.517485    4898 cpu_manager.go:414] "containerMap: removing container" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy"
Nov 02 07:53:28.517546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.517493    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76acab59-e07d-49ae-9075-0accdbe6e5b1" containerName="istio-proxy"
Nov 02 07:53:28.517716 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.517618    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerName="istio-proxy"
Nov 02 07:53:28.521998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.521975    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:28.531597 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.531122    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:28.538908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.538734    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"]
Nov 02 07:53:28.640567 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.640512    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-openshift-ca\") pod \"7d625389-08c3-4009-beb8-68cc7832967f\" (UID: \"7d625389-08c3-4009-beb8-68cc7832967f\") "
Nov 02 07:53:28.640567 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.640561    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-xks-ca\") pod \"7d625389-08c3-4009-beb8-68cc7832967f\" (UID: \"7d625389-08c3-4009-beb8-68cc7832967f\") "
Nov 02 07:53:28.640819 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.640760    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d625389-08c3-4009-beb8-68cc7832967f-kube-api-access-pcxfh\" (UniqueName: \"kubernetes.io/projected/7d625389-08c3-4009-beb8-68cc7832967f-kube-api-access-pcxfh\") pod \"7d625389-08c3-4009-beb8-68cc7832967f\" (UID: \"7d625389-08c3-4009-beb8-68cc7832967f\") "
Nov 02 07:53:28.641354 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.641318    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-693b3763-tj52l\" (UID: \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:28.641517 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.641446    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctqq4\" (UniqueName: \"kubernetes.io/projected/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-kube-api-access-ctqq4\") pod \"maas-api-revoke-sub-693b3763-tj52l\" (UID: \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:28.641590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.641521    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-693b3763-tj52l\" (UID: \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:28.643956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.643929    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-xks-ca" pod "7d625389-08c3-4009-beb8-68cc7832967f" (UID: "7d625389-08c3-4009-beb8-68cc7832967f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:53:28.644250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.644229    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d625389-08c3-4009-beb8-68cc7832967f-kube-api-access-pcxfh" pod "7d625389-08c3-4009-beb8-68cc7832967f" (UID: "7d625389-08c3-4009-beb8-68cc7832967f"). InnerVolumeSpecName "kube-api-access-pcxfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:28.653108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.653070    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-openshift-ca" pod "7d625389-08c3-4009-beb8-68cc7832967f" (UID: "7d625389-08c3-4009-beb8-68cc7832967f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:28.742437 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.742394    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-693b3763-tj52l\" (UID: \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:28.742610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.742467    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctqq4\" (UniqueName: \"kubernetes.io/projected/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-kube-api-access-ctqq4\") pod \"maas-api-revoke-sub-693b3763-tj52l\" (UID: \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:28.742610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.742530    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-693b3763-tj52l\" (UID: \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:28.742763 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.742680    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcxfh\" (UniqueName: \"kubernetes.io/projected/7d625389-08c3-4009-beb8-68cc7832967f-kube-api-access-pcxfh\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:28.742763 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.742700    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:28.742763 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.742714    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d625389-08c3-4009-beb8-68cc7832967f-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:28.743240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.743217    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-693b3763-tj52l\" (UID: \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:28.744859 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.744836    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-693b3763-tj52l\" (UID: \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:28.789376 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.789285    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctqq4\" (UniqueName: \"kubernetes.io/projected/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-kube-api-access-ctqq4\") pod \"maas-api-revoke-sub-693b3763-tj52l\" (UID: \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:28.850273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:28.850240    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:29.189855 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:29.189820    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"]
Nov 02 07:53:29.190028 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:53:29.189981    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ace2b5a_db26_4bc6_9670_6a790d0f8b69.slice/crio-fbd8b93abc45917cb79c536f95b9f137d7b981ae93815001c88f253b935c6ae7 WatchSource:0}: Error finding container fbd8b93abc45917cb79c536f95b9f137d7b981ae93815001c88f253b935c6ae7: Status 404 returned error can't find the container with id fbd8b93abc45917cb79c536f95b9f137d7b981ae93815001c88f253b935c6ae7
Nov 02 07:53:29.242653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:29.242629    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"
Nov 02 07:53:29.242653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:29.242641    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv" event={"ID":"7d625389-08c3-4009-beb8-68cc7832967f","Type":"ContainerDied","Data":"24b2485ac90d953791bedca9e49e19a902ea33f3655750ea3f652da85560f9fb"}
Nov 02 07:53:29.242822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:29.242705    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24b2485ac90d953791bedca9e49e19a902ea33f3655750ea3f652da85560f9fb"
Nov 02 07:53:29.244082 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:29.244057    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l" event={"ID":"1ace2b5a-db26-4bc6-9670-6a790d0f8b69","Type":"ContainerStarted","Data":"fbd8b93abc45917cb79c536f95b9f137d7b981ae93815001c88f253b935c6ae7"}
Nov 02 07:53:30.305985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:30.305945    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l" event={"ID":"1ace2b5a-db26-4bc6-9670-6a790d0f8b69","Type":"ContainerStarted","Data":"f1c16e6d0869ead83779fe5ca9e734ed688660ea9ec84add3a559d09bfbc1e07"}
Nov 02 07:53:30.348675 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:30.348624    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l" podStartSLOduration=2.348610377 podStartE2EDuration="2.348610377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:30.346022063 +0000 UTC m=+1761.036349022" watchObservedRunningTime="2026-11-02 07:53:30.348610377 +0000 UTC m=+1761.038937341"
Nov 02 07:53:31.316438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:31.316403    4898 generic.go:356] "Generic (PLEG): container finished" podID="1ace2b5a-db26-4bc6-9670-6a790d0f8b69" containerID="f1c16e6d0869ead83779fe5ca9e734ed688660ea9ec84add3a559d09bfbc1e07" exitCode=0
Nov 02 07:53:31.316956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:31.316497    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l" event={"ID":"1ace2b5a-db26-4bc6-9670-6a790d0f8b69","Type":"ContainerDied","Data":"f1c16e6d0869ead83779fe5ca9e734ed688660ea9ec84add3a559d09bfbc1e07"}
Nov 02 07:53:32.634074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:32.634004    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:32.732142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:32.731833    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-xks-ca\") pod \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\" (UID: \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\") "
Nov 02 07:53:32.732142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:32.731945    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-openshift-ca\") pod \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\" (UID: \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\") "
Nov 02 07:53:32.732142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:32.732007    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-kube-api-access-ctqq4\" (UniqueName: \"kubernetes.io/projected/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-kube-api-access-ctqq4\") pod \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\" (UID: \"1ace2b5a-db26-4bc6-9670-6a790d0f8b69\") "
Nov 02 07:53:32.732472 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:32.732398    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-openshift-ca" pod "1ace2b5a-db26-4bc6-9670-6a790d0f8b69" (UID: "1ace2b5a-db26-4bc6-9670-6a790d0f8b69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:32.732702 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:32.732679    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:32.734858 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:32.734829    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-xks-ca" pod "1ace2b5a-db26-4bc6-9670-6a790d0f8b69" (UID: "1ace2b5a-db26-4bc6-9670-6a790d0f8b69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:53:32.734974 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:32.734861    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-kube-api-access-ctqq4" pod "1ace2b5a-db26-4bc6-9670-6a790d0f8b69" (UID: "1ace2b5a-db26-4bc6-9670-6a790d0f8b69"). InnerVolumeSpecName "kube-api-access-ctqq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:32.833681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:32.833636    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctqq4\" (UniqueName: \"kubernetes.io/projected/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-kube-api-access-ctqq4\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:32.833681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:32.833675    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ace2b5a-db26-4bc6-9670-6a790d0f8b69-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:33.335479 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:33.335448    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"
Nov 02 07:53:33.335659 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:33.335480    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l" event={"ID":"1ace2b5a-db26-4bc6-9670-6a790d0f8b69","Type":"ContainerDied","Data":"fbd8b93abc45917cb79c536f95b9f137d7b981ae93815001c88f253b935c6ae7"}
Nov 02 07:53:33.335659 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:33.335530    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbd8b93abc45917cb79c536f95b9f137d7b981ae93815001c88f253b935c6ae7"
Nov 02 07:53:37.411661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:37.411012    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": read tcp 10.132.0.2:43794->10.132.0.204:15021: read: connection reset by peer" start-of-body=
Nov 02 07:53:37.411661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:37.411070    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": read tcp 10.132.0.2:43794->10.132.0.204:15021: read: connection reset by peer"
Nov 02 07:53:38.391706 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:38.391673    4898 generic.go:356] "Generic (PLEG): container finished" podID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerID="4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a" exitCode=0
Nov 02 07:53:38.391882 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:38.391747    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" event={"ID":"0d446180-273b-4b72-9dcf-e0979ddbe681","Type":"ContainerDied","Data":"4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a"}
Nov 02 07:53:38.392251 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:38.392234    4898 scope.go:122] "RemoveContainer" containerID="4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a"
Nov 02 07:53:38.394095 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:38.394063    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:38.394217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:38.394140    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:38.394217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:38.394176    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:39.402667 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:39.402628    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" event={"ID":"0d446180-273b-4b72-9dcf-e0979ddbe681","Type":"ContainerStarted","Data":"596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba"}
Nov 02 07:53:39.830094 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:39.830001    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:53:39.830286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:39.830257    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 07:53:39.830362 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:39.830336    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 07:53:40.204389 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.204352    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"]
Nov 02 07:53:40.204952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.204850    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796" podUID="233e4b67-1d34-446a-b44a-6ee2f2731df5" containerName="istio-proxy" containerID="cri-o://402c54bed947226c4a24289c87000297d9610137a9bf3e2bf5309b574ffe792d" gracePeriod=30
Nov 02 07:53:40.209416 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.209395    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"]
Nov 02 07:53:40.209997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.209826    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd" podUID="d2b4be44-0446-4e93-b48d-63e17726be91" containerName="istio-proxy" containerID="cri-o://a52f02d75cb9226dfa6a67f86fc1cf58657707685cdadce1f3f881fe07041f4c" gracePeriod=30
Nov 02 07:53:40.215852 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.215828    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"]
Nov 02 07:53:40.216033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.216013    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77" podUID="27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" containerName="istio-proxy" containerID="cri-o://3444669bd923d6547eb769e4191ee5a9ae011b22767ce00c5bb21ebae4a20fdd" gracePeriod=30
Nov 02 07:53:40.630512 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.630418    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"]
Nov 02 07:53:40.637271 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.637240    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerName="istio-proxy"
Nov 02 07:53:40.637501 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.637480    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerName="istio-proxy"
Nov 02 07:53:40.637636 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.637622    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7d625389-08c3-4009-beb8-68cc7832967f" containerName="revoke-keys"
Nov 02 07:53:40.637740 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.637726    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d625389-08c3-4009-beb8-68cc7832967f" containerName="revoke-keys"
Nov 02 07:53:40.637877 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.637865    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1ace2b5a-db26-4bc6-9670-6a790d0f8b69" containerName="revoke-keys"
Nov 02 07:53:40.638443 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.638424    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ace2b5a-db26-4bc6-9670-6a790d0f8b69" containerName="revoke-keys"
Nov 02 07:53:40.638755 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.638741    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d625389-08c3-4009-beb8-68cc7832967f" containerName="revoke-keys"
Nov 02 07:53:40.638860 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.638850    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ace2b5a-db26-4bc6-9670-6a790d0f8b69" containerName="revoke-keys"
Nov 02 07:53:40.638969 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.638958    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a9a6c9-7420-428e-a684-8dba457ecae9" containerName="istio-proxy"
Nov 02 07:53:40.647302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.647275    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.659726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.657996    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"]
Nov 02 07:53:40.662436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.662392    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-32ea65-openshift-default-dockercfg-pfmtz\""
Nov 02 07:53:40.737837 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.737800    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-envoy\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.738149 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.738129    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-socket\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.738346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.738332    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g2ds\" (UniqueName: \"kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-kube-api-access-4g2ds\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.738448 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.738435    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-certs\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.738561 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.738550    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c33009d-0656-4873-9be6-968f615c990d-istio-podinfo\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.738749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.738734    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-credential-socket\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.738956 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.738938    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-istio-token\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.740411 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.740370    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c33009d-0656-4873-9be6-968f615c990d-istiod-ca-cert\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.741044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.741020    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-data\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.833523 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.833485    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 07:53:40.833813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.833785    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 07:53:40.843280 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.843247    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g2ds\" (UniqueName: \"kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-kube-api-access-4g2ds\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.843485 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.843472    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-certs\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.843573 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.843564    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c33009d-0656-4873-9be6-968f615c990d-istio-podinfo\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.843706 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.843698    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-credential-socket\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.843781 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.843773    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-istio-token\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.843850 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.843840    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c33009d-0656-4873-9be6-968f615c990d-istiod-ca-cert\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.843985 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.843973    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-data\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.844206 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.844187    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-envoy\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.844362 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.844344    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-socket\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.844457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.844440    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-credential-socket\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.844561 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.844017    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-certs\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.844784 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.844762    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-socket\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.844857 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.844796    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c33009d-0656-4873-9be6-968f615c990d-istiod-ca-cert\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.845127 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.845104    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-data\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.846601 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.846558    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-envoy\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.847041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.847020    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c33009d-0656-4873-9be6-968f615c990d-istio-podinfo\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.885225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.885157    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g2ds\" (UniqueName: \"kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-kube-api-access-4g2ds\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:40.898277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:40.898251    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-istio-token\") pod \"e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") " pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:41.015427 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:41.015382    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:41.391504 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:41.391471    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"]
Nov 02 07:53:41.393540 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:53:41.393509    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c33009d_0656_4873_9be6_968f615c990d.slice/crio-d0992753ec3ca6d8e5319324cfe2d363441274e23a18782030b36258084ca5df WatchSource:0}: Error finding container d0992753ec3ca6d8e5319324cfe2d363441274e23a18782030b36258084ca5df: Status 404 returned error can't find the container with id d0992753ec3ca6d8e5319324cfe2d363441274e23a18782030b36258084ca5df
Nov 02 07:53:41.396475 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:41.396117    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:41.396475 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:41.396208    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:41.396475 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:41.396240    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:53:41.494649 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:41.494612    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" event={"ID":"2c33009d-0656-4873-9be6-968f615c990d","Type":"ContainerStarted","Data":"d0992753ec3ca6d8e5319324cfe2d363441274e23a18782030b36258084ca5df"}
Nov 02 07:53:41.830766 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:41.830715    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 07:53:41.831273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:41.830792    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 07:53:42.522939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:42.521582    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" event={"ID":"2c33009d-0656-4873-9be6-968f615c990d","Type":"ContainerStarted","Data":"897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b"}
Nov 02 07:53:42.602204 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:42.602150    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" podStartSLOduration=2.602135916 podStartE2EDuration="2.602135916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:42.582335944 +0000 UTC m=+1773.272662934" watchObservedRunningTime="2026-11-02 07:53:42.602135916 +0000 UTC m=+1773.292462878"
Nov 02 07:53:42.844781 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:42.836352    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 07:53:42.844781 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:42.836437    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 07:53:43.017013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:43.016956    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:43.022564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:43.022538    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:43.529279 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:43.529244    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:43.530136 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:43.530111    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:53:43.830487 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:43.830408    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 07:53:43.830638 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:43.830479    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 07:53:44.834200 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:44.834169    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:53:44.834683 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:44.834670    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:53:44.835565 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:44.835545    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 07:53:45.005009 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:53:45.004964    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:53:45.581393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.581159    4898 generic.go:356] "Generic (PLEG): container finished" podID="d2b4be44-0446-4e93-b48d-63e17726be91" containerID="a52f02d75cb9226dfa6a67f86fc1cf58657707685cdadce1f3f881fe07041f4c" exitCode=0
Nov 02 07:53:45.584925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.584729    4898 generic.go:356] "Generic (PLEG): container finished" podID="27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" containerID="3444669bd923d6547eb769e4191ee5a9ae011b22767ce00c5bb21ebae4a20fdd" exitCode=0
Nov 02 07:53:45.589469 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.588196    4898 generic.go:356] "Generic (PLEG): container finished" podID="233e4b67-1d34-446a-b44a-6ee2f2731df5" containerID="402c54bed947226c4a24289c87000297d9610137a9bf3e2bf5309b574ffe792d" exitCode=0
Nov 02 07:53:45.627872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.627753    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd" event={"ID":"d2b4be44-0446-4e93-b48d-63e17726be91","Type":"ContainerDied","Data":"a52f02d75cb9226dfa6a67f86fc1cf58657707685cdadce1f3f881fe07041f4c"}
Nov 02 07:53:45.627872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.627807    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77" event={"ID":"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c","Type":"ContainerDied","Data":"3444669bd923d6547eb769e4191ee5a9ae011b22767ce00c5bb21ebae4a20fdd"}
Nov 02 07:53:45.627872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.627829    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796" event={"ID":"233e4b67-1d34-446a-b44a-6ee2f2731df5","Type":"ContainerDied","Data":"402c54bed947226c4a24289c87000297d9610137a9bf3e2bf5309b574ffe792d"}
Nov 02 07:53:45.723391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.723224    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:45.821214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.821183    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:53:45.845227 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.845039    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:45.854500 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.854471    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-podinfo\") pod \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") "
Nov 02 07:53:45.854642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.854516    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istiod-ca-cert\") pod \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") "
Nov 02 07:53:45.854642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.854591    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-token\" (UniqueName: \"kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-token\") pod \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") "
Nov 02 07:53:45.854773 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.854750    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-credential-socket\") pod \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") "
Nov 02 07:53:45.854832 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.854800    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-certs\") pod \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") "
Nov 02 07:53:45.854923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.854842    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-envoy\") pod \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") "
Nov 02 07:53:45.854923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.854874    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-socket\") pod \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") "
Nov 02 07:53:45.855160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.854975    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-data\") pod \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") "
Nov 02 07:53:45.855160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.855091    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-certs" pod "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" (UID: "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.855160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.855113    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istiod-ca-cert" pod "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" (UID: "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:45.855160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.855146    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-kube-api-access-mkgk6\" (UniqueName: \"kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-kube-api-access-mkgk6\") pod \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\" (UID: \"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c\") "
Nov 02 07:53:45.855368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.855344    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-socket" pod "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" (UID: "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.855537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.855514    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-data" pod "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" (UID: "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.855697 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.855673    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-credential-socket" pod "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" (UID: "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.855911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.855859    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.856003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.855911    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.856003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.855928    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.856003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.855954    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.856003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.855967    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.858945 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.858918    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-token" pod "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" (UID: "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:45.859236 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.859200    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-envoy" pod "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" (UID: "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.862010 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.861982    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-kube-api-access-mkgk6" pod "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" (UID: "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c"). InnerVolumeSpecName "kube-api-access-mkgk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:45.864826 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.864800    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-podinfo" pod "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" (UID: "27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:53:45.956434 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.956394    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-kube-api-access-v4gqn\" (UniqueName: \"kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-kube-api-access-v4gqn\") pod \"d2b4be44-0446-4e93-b48d-63e17726be91\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") "
Nov 02 07:53:45.956628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.956467    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-podinfo\") pod \"233e4b67-1d34-446a-b44a-6ee2f2731df5\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") "
Nov 02 07:53:45.956628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.956561    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-certs\") pod \"233e4b67-1d34-446a-b44a-6ee2f2731df5\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") "
Nov 02 07:53:45.956758 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.956650    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-credential-socket\") pod \"233e4b67-1d34-446a-b44a-6ee2f2731df5\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") "
Nov 02 07:53:45.956758 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.956685    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-envoy\") pod \"233e4b67-1d34-446a-b44a-6ee2f2731df5\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") "
Nov 02 07:53:45.956758 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.956715    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/233e4b67-1d34-446a-b44a-6ee2f2731df5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233e4b67-1d34-446a-b44a-6ee2f2731df5-istiod-ca-cert\") pod \"233e4b67-1d34-446a-b44a-6ee2f2731df5\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") "
Nov 02 07:53:45.956758 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.956747    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b4be44-0446-4e93-b48d-63e17726be91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b4be44-0446-4e93-b48d-63e17726be91-istiod-ca-cert\") pod \"d2b4be44-0446-4e93-b48d-63e17726be91\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") "
Nov 02 07:53:45.956998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.956793    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-socket\") pod \"233e4b67-1d34-446a-b44a-6ee2f2731df5\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") "
Nov 02 07:53:45.956998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.956845    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-certs\") pod \"d2b4be44-0446-4e93-b48d-63e17726be91\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") "
Nov 02 07:53:45.956998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.956922    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-data\") pod \"233e4b67-1d34-446a-b44a-6ee2f2731df5\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") "
Nov 02 07:53:45.956998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.956977    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-credential-socket\") pod \"d2b4be44-0446-4e93-b48d-63e17726be91\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") "
Nov 02 07:53:45.957270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.957050    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-kube-api-access-qczgc\" (UniqueName: \"kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-kube-api-access-qczgc\") pod \"233e4b67-1d34-446a-b44a-6ee2f2731df5\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") "
Nov 02 07:53:45.957270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.957090    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-data\") pod \"d2b4be44-0446-4e93-b48d-63e17726be91\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") "
Nov 02 07:53:45.957270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.957144    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2b4be44-0446-4e93-b48d-63e17726be91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b4be44-0446-4e93-b48d-63e17726be91-istio-podinfo\") pod \"d2b4be44-0446-4e93-b48d-63e17726be91\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") "
Nov 02 07:53:45.957270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.957180    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-envoy\") pod \"d2b4be44-0446-4e93-b48d-63e17726be91\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") "
Nov 02 07:53:45.957270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.957245    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-istio-token\" (UniqueName: \"kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-istio-token\") pod \"d2b4be44-0446-4e93-b48d-63e17726be91\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") "
Nov 02 07:53:45.957552 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.957272    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b4be44-0446-4e93-b48d-63e17726be91-istiod-ca-cert" pod "d2b4be44-0446-4e93-b48d-63e17726be91" (UID: "d2b4be44-0446-4e93-b48d-63e17726be91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:45.957552 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.957484    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-credential-socket" pod "233e4b67-1d34-446a-b44a-6ee2f2731df5" (UID: "233e4b67-1d34-446a-b44a-6ee2f2731df5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.957663 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.957561    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-credential-socket" pod "d2b4be44-0446-4e93-b48d-63e17726be91" (UID: "d2b4be44-0446-4e93-b48d-63e17726be91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.957666    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233e4b67-1d34-446a-b44a-6ee2f2731df5-istiod-ca-cert" pod "233e4b67-1d34-446a-b44a-6ee2f2731df5" (UID: "233e4b67-1d34-446a-b44a-6ee2f2731df5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.957741    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-socket" pod "233e4b67-1d34-446a-b44a-6ee2f2731df5" (UID: "233e4b67-1d34-446a-b44a-6ee2f2731df5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.957816    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-certs" pod "233e4b67-1d34-446a-b44a-6ee2f2731df5" (UID: "233e4b67-1d34-446a-b44a-6ee2f2731df5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.957940    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-certs" pod "d2b4be44-0446-4e93-b48d-63e17726be91" (UID: "d2b4be44-0446-4e93-b48d-63e17726be91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.958210    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-token\" (UniqueName: \"kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-token\") pod \"233e4b67-1d34-446a-b44a-6ee2f2731df5\" (UID: \"233e4b67-1d34-446a-b44a-6ee2f2731df5\") "
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.958274    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-socket\") pod \"d2b4be44-0446-4e93-b48d-63e17726be91\" (UID: \"d2b4be44-0446-4e93-b48d-63e17726be91\") "
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959038    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959060    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959075    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959091    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/233e4b67-1d34-446a-b44a-6ee2f2731df5-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959106    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b4be44-0446-4e93-b48d-63e17726be91-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959120    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959134    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959148    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959163    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959180    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkgk6\" (UniqueName: \"kubernetes.io/projected/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-kube-api-access-mkgk6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959196    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959256    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-data" pod "d2b4be44-0446-4e93-b48d-63e17726be91" (UID: "d2b4be44-0446-4e93-b48d-63e17726be91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.959691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.959411    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-data" pod "233e4b67-1d34-446a-b44a-6ee2f2731df5" (UID: "233e4b67-1d34-446a-b44a-6ee2f2731df5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.960680 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.960105    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-kube-api-access-v4gqn" pod "d2b4be44-0446-4e93-b48d-63e17726be91" (UID: "d2b4be44-0446-4e93-b48d-63e17726be91"). InnerVolumeSpecName "kube-api-access-v4gqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:45.960680 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.960207    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2b4be44-0446-4e93-b48d-63e17726be91-istio-podinfo" pod "d2b4be44-0446-4e93-b48d-63e17726be91" (UID: "d2b4be44-0446-4e93-b48d-63e17726be91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:53:45.960680 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.960467    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-envoy" pod "233e4b67-1d34-446a-b44a-6ee2f2731df5" (UID: "233e4b67-1d34-446a-b44a-6ee2f2731df5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.960680 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.960549    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-podinfo" pod "233e4b67-1d34-446a-b44a-6ee2f2731df5" (UID: "233e4b67-1d34-446a-b44a-6ee2f2731df5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:53:45.960934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.960901    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-socket" pod "d2b4be44-0446-4e93-b48d-63e17726be91" (UID: "d2b4be44-0446-4e93-b48d-63e17726be91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.962297 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.962266    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-kube-api-access-qczgc" pod "233e4b67-1d34-446a-b44a-6ee2f2731df5" (UID: "233e4b67-1d34-446a-b44a-6ee2f2731df5"). InnerVolumeSpecName "kube-api-access-qczgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:45.962406 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.962294    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-istio-token" pod "d2b4be44-0446-4e93-b48d-63e17726be91" (UID: "d2b4be44-0446-4e93-b48d-63e17726be91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:45.962814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.962788    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-envoy" pod "d2b4be44-0446-4e93-b48d-63e17726be91" (UID: "d2b4be44-0446-4e93-b48d-63e17726be91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:53:45.963407 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:45.963371    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-token" pod "233e4b67-1d34-446a-b44a-6ee2f2731df5" (UID: "233e4b67-1d34-446a-b44a-6ee2f2731df5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:53:46.061111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.061065    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.061111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.061107    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.061111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.061121    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.061446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.061135    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qczgc\" (UniqueName: \"kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-kube-api-access-qczgc\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.061446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.061148    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.061446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.061161    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b4be44-0446-4e93-b48d-63e17726be91-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.061446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.061174    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.061446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.061187    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.061446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.061199    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/233e4b67-1d34-446a-b44a-6ee2f2731df5-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.061446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.061212    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b4be44-0446-4e93-b48d-63e17726be91-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.061446 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.061224    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4gqn\" (UniqueName: \"kubernetes.io/projected/d2b4be44-0446-4e93-b48d-63e17726be91-kube-api-access-v4gqn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:53:46.622319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.620941    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd" event={"ID":"d2b4be44-0446-4e93-b48d-63e17726be91","Type":"ContainerDied","Data":"45297959d0dbf5647923781ced759f3d53993bcafb90808f8706a7b08e407af4"}
Nov 02 07:53:46.622319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.621014    4898 scope.go:122] "RemoveContainer" containerID="a52f02d75cb9226dfa6a67f86fc1cf58657707685cdadce1f3f881fe07041f4c"
Nov 02 07:53:46.622319 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.620958    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"
Nov 02 07:53:46.625196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.624812    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"
Nov 02 07:53:46.625196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.624849    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77" event={"ID":"27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c","Type":"ContainerDied","Data":"b98eb3740960a37b47c837a47fa47eb4b0c0a8c4c3f7fb8f11be3d5afed92073"}
Nov 02 07:53:46.631940 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.631181    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"
Nov 02 07:53:46.632941 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.632917    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796" event={"ID":"233e4b67-1d34-446a-b44a-6ee2f2731df5","Type":"ContainerDied","Data":"1f5f9e3695bf259dae907664696a8b5b5c19537b98375c68a689fd4c251861f2"}
Nov 02 07:53:46.638615 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.638598    4898 scope.go:122] "RemoveContainer" containerID="3444669bd923d6547eb769e4191ee5a9ae011b22767ce00c5bb21ebae4a20fdd"
Nov 02 07:53:46.652917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.652856    4898 scope.go:122] "RemoveContainer" containerID="402c54bed947226c4a24289c87000297d9610137a9bf3e2bf5309b574ffe792d"
Nov 02 07:53:46.707705 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.707676    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"]
Nov 02 07:53:46.718884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.718854    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-t7mrd"]
Nov 02 07:53:46.754106 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.754079    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"]
Nov 02 07:53:46.760007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.759980    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-j8796"]
Nov 02 07:53:46.798692 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.798631    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"]
Nov 02 07:53:46.803235 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:46.803210    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9ab4d2-openshift-default-9f4954bfc-9sm77"]
Nov 02 07:53:47.469457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:47.469411    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233e4b67-1d34-446a-b44a-6ee2f2731df5" path="/var/lib/kubelet/pods/233e4b67-1d34-446a-b44a-6ee2f2731df5/volumes"
Nov 02 07:53:47.470133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:47.470115    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" path="/var/lib/kubelet/pods/27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c/volumes"
Nov 02 07:53:47.470583 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:47.470568    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b4be44-0446-4e93-b48d-63e17726be91" path="/var/lib/kubelet/pods/d2b4be44-0446-4e93-b48d-63e17726be91/volumes"
Nov 02 07:53:48.210246 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.210212    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"]
Nov 02 07:53:48.211018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.210973    4898 cpu_manager.go:414] "containerMap: removing container" podUID="27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" containerName="istio-proxy"
Nov 02 07:53:48.211018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.210993    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" containerName="istio-proxy"
Nov 02 07:53:48.211018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.211010    4898 cpu_manager.go:414] "containerMap: removing container" podUID="233e4b67-1d34-446a-b44a-6ee2f2731df5" containerName="istio-proxy"
Nov 02 07:53:48.211018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.211015    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233e4b67-1d34-446a-b44a-6ee2f2731df5" containerName="istio-proxy"
Nov 02 07:53:48.211018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.211026    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d2b4be44-0446-4e93-b48d-63e17726be91" containerName="istio-proxy"
Nov 02 07:53:48.211355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.211034    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b4be44-0446-4e93-b48d-63e17726be91" containerName="istio-proxy"
Nov 02 07:53:48.211355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.211202    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b4d4f7-1bc1-42e2-8add-e43fc90e8b4c" containerName="istio-proxy"
Nov 02 07:53:48.211355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.211217    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="233e4b67-1d34-446a-b44a-6ee2f2731df5" containerName="istio-proxy"
Nov 02 07:53:48.211355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.211223    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b4be44-0446-4e93-b48d-63e17726be91" containerName="istio-proxy"
Nov 02 07:53:48.216650 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.216630    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"
Nov 02 07:53:48.234998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.234972    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-32ea65-dockercfg-htcsh\""
Nov 02 07:53:48.237212 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.237192    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-32ea65\""
Nov 02 07:53:48.238875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.238854    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"]
Nov 02 07:53:48.332900 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.332856    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt\" (UID: \"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"
Nov 02 07:53:48.333071 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.332982    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dhhh\" (UniqueName: \"kubernetes.io/projected/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-kube-api-access-8dhhh\") pod \"payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt\" (UID: \"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"
Nov 02 07:53:48.434015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.433983    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt\" (UID: \"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"
Nov 02 07:53:48.434187 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.434043    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dhhh\" (UniqueName: \"kubernetes.io/projected/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-kube-api-access-8dhhh\") pod \"payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt\" (UID: \"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"
Nov 02 07:53:48.434741 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.434716    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt\" (UID: \"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"
Nov 02 07:53:48.485961 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.484804    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dhhh\" (UniqueName: \"kubernetes.io/projected/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-kube-api-access-8dhhh\") pod \"payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt\" (UID: \"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"
Nov 02 07:53:48.536913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:48.536149    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"
Nov 02 07:53:49.114055 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:53:49.114019    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5379dd80_6ad8_4b2a_a49c_e012dfc0fab6.slice/crio-76c2a111b72f2ee247a4cc673962240b5fc210afe311fb04dbd4e74c7d75177a WatchSource:0}: Error finding container 76c2a111b72f2ee247a4cc673962240b5fc210afe311fb04dbd4e74c7d75177a: Status 404 returned error can't find the container with id 76c2a111b72f2ee247a4cc673962240b5fc210afe311fb04dbd4e74c7d75177a
Nov 02 07:53:49.116119 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.116093    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"]
Nov 02 07:53:49.149343 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.149318    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"]
Nov 02 07:53:49.155559 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.155537    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"
Nov 02 07:53:49.170858 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.170834    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-56bbb6\""
Nov 02 07:53:49.171033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.170879    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-56bbb6-dockercfg-6wqxs\""
Nov 02 07:53:49.181133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.175696    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"]
Nov 02 07:53:49.199193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.198883    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13afe874-6c21-463a-b1fb-829e7d104bb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw\" (UID: \"13afe874-6c21-463a-b1fb-829e7d104bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"
Nov 02 07:53:49.199193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.198961    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2q2b\" (UniqueName: \"kubernetes.io/projected/13afe874-6c21-463a-b1fb-829e7d104bb8-kube-api-access-q2q2b\") pod \"payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw\" (UID: \"13afe874-6c21-463a-b1fb-829e7d104bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"
Nov 02 07:53:49.240553 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.240510    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"]
Nov 02 07:53:49.244978 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.244956    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"
Nov 02 07:53:49.272861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.272831    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"]
Nov 02 07:53:49.374046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.364924    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wst85\" (UniqueName: \"kubernetes.io/projected/2b3467c1-86b7-479b-8bc3-89e7b0473d89-kube-api-access-wst85\") pod \"payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h\" (UID: \"2b3467c1-86b7-479b-8bc3-89e7b0473d89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"
Nov 02 07:53:49.374046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.365062    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3467c1-86b7-479b-8bc3-89e7b0473d89-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h\" (UID: \"2b3467c1-86b7-479b-8bc3-89e7b0473d89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"
Nov 02 07:53:49.374046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.365113    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13afe874-6c21-463a-b1fb-829e7d104bb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw\" (UID: \"13afe874-6c21-463a-b1fb-829e7d104bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"
Nov 02 07:53:49.374046 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.365154    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2q2b\" (UniqueName: \"kubernetes.io/projected/13afe874-6c21-463a-b1fb-829e7d104bb8-kube-api-access-q2q2b\") pod \"payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw\" (UID: \"13afe874-6c21-463a-b1fb-829e7d104bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"
Nov 02 07:53:49.402571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.402525    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13afe874-6c21-463a-b1fb-829e7d104bb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw\" (UID: \"13afe874-6c21-463a-b1fb-829e7d104bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"
Nov 02 07:53:49.480992 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.480936    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wst85\" (UniqueName: \"kubernetes.io/projected/2b3467c1-86b7-479b-8bc3-89e7b0473d89-kube-api-access-wst85\") pod \"payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h\" (UID: \"2b3467c1-86b7-479b-8bc3-89e7b0473d89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"
Nov 02 07:53:49.481190 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.481085    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3467c1-86b7-479b-8bc3-89e7b0473d89-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h\" (UID: \"2b3467c1-86b7-479b-8bc3-89e7b0473d89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"
Nov 02 07:53:49.482356 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.482325    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3467c1-86b7-479b-8bc3-89e7b0473d89-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h\" (UID: \"2b3467c1-86b7-479b-8bc3-89e7b0473d89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"
Nov 02 07:53:49.562914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.551523    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2q2b\" (UniqueName: \"kubernetes.io/projected/13afe874-6c21-463a-b1fb-829e7d104bb8-kube-api-access-q2q2b\") pod \"payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw\" (UID: \"13afe874-6c21-463a-b1fb-829e7d104bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"
Nov 02 07:53:49.589157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.589121    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wst85\" (UniqueName: \"kubernetes.io/projected/2b3467c1-86b7-479b-8bc3-89e7b0473d89-kube-api-access-wst85\") pod \"payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h\" (UID: \"2b3467c1-86b7-479b-8bc3-89e7b0473d89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"
Nov 02 07:53:49.739080 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.739035    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt" event={"ID":"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6","Type":"ContainerStarted","Data":"e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d"}
Nov 02 07:53:49.739274 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.739093    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt" event={"ID":"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6","Type":"ContainerStarted","Data":"76c2a111b72f2ee247a4cc673962240b5fc210afe311fb04dbd4e74c7d75177a"}
Nov 02 07:53:49.739338 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.739312    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"
Nov 02 07:53:49.781341 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.781286    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt" podStartSLOduration=1.78126794 podStartE2EDuration="1.78126794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:49.779335365 +0000 UTC m=+1780.469662328" watchObservedRunningTime="2026-11-02 07:53:49.78126794 +0000 UTC m=+1780.471594931"
Nov 02 07:53:49.801922 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.801877    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"
Nov 02 07:53:49.868065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:49.868029    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"
Nov 02 07:53:50.139423 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:50.136814    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"]
Nov 02 07:53:50.144458 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:53:50.144424    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13afe874_6c21_463a_b1fb_829e7d104bb8.slice/crio-dc9e9e89b4bb9fd373546affc95181a19afeeffcdaec6ed62f6d60c237a1d3c4 WatchSource:0}: Error finding container dc9e9e89b4bb9fd373546affc95181a19afeeffcdaec6ed62f6d60c237a1d3c4: Status 404 returned error can't find the container with id dc9e9e89b4bb9fd373546affc95181a19afeeffcdaec6ed62f6d60c237a1d3c4
Nov 02 07:53:50.357321 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:53:50.357289    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3467c1_86b7_479b_8bc3_89e7b0473d89.slice/crio-25982b38fdb72201774386519e5e51f00834b47a41c37d9ae2532cf2f61f4d18 WatchSource:0}: Error finding container 25982b38fdb72201774386519e5e51f00834b47a41c37d9ae2532cf2f61f4d18: Status 404 returned error can't find the container with id 25982b38fdb72201774386519e5e51f00834b47a41c37d9ae2532cf2f61f4d18
Nov 02 07:53:50.362994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:50.362967    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"]
Nov 02 07:53:50.750140 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:50.750097    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h" event={"ID":"2b3467c1-86b7-479b-8bc3-89e7b0473d89","Type":"ContainerStarted","Data":"6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7"}
Nov 02 07:53:50.750611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:50.750202    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"
Nov 02 07:53:50.750611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:50.750231    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h" event={"ID":"2b3467c1-86b7-479b-8bc3-89e7b0473d89","Type":"ContainerStarted","Data":"25982b38fdb72201774386519e5e51f00834b47a41c37d9ae2532cf2f61f4d18"}
Nov 02 07:53:50.752036 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:50.752004    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw" event={"ID":"13afe874-6c21-463a-b1fb-829e7d104bb8","Type":"ContainerStarted","Data":"0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb"}
Nov 02 07:53:50.752214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:50.752057    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw" event={"ID":"13afe874-6c21-463a-b1fb-829e7d104bb8","Type":"ContainerStarted","Data":"dc9e9e89b4bb9fd373546affc95181a19afeeffcdaec6ed62f6d60c237a1d3c4"}
Nov 02 07:53:50.752284 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:50.752205    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"
Nov 02 07:53:50.814913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:50.801407    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h" podStartSLOduration=1.8013891549999999 podStartE2EDuration="1.801389155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:50.791225561 +0000 UTC m=+1781.481552539" watchObservedRunningTime="2026-11-02 07:53:50.801389155 +0000 UTC m=+1781.491716120"
Nov 02 07:53:50.845281 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:53:50.845226    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw" podStartSLOduration=1.845208884 podStartE2EDuration="1.845208884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:53:50.842074562 +0000 UTC m=+1781.532401519" watchObservedRunningTime="2026-11-02 07:53:50.845208884 +0000 UTC m=+1781.535535849"
Nov 02 07:54:00.755364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:00.755334    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"
Nov 02 07:54:01.766533 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:01.766503    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"
Nov 02 07:54:01.767065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:01.766569    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"
Nov 02 07:54:10.152979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:10.152246    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:54:10.152979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:10.152915    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:54:14.991015 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:54:14.990976    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:54:18.128833 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.128796    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"]
Nov 02 07:54:18.133279 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.133259    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.155710 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.155492    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"]
Nov 02 07:54:18.257934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.257875    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.258117 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.258035    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.258117 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.258092    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.258206 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.258149    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.258303 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.258275    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.258356 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.258322    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7198de3d-596e-4e43-a71c-d1dcf8deea73-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.258408 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.258393    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85nfd\" (UniqueName: \"kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-kube-api-access-85nfd\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.258501 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.258486    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.258631 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.258614    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.360424 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.360386    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.360424 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.360421    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7198de3d-596e-4e43-a71c-d1dcf8deea73-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.360831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.360446    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85nfd\" (UniqueName: \"kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-kube-api-access-85nfd\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.360831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.360484    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.360831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.360555    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.360831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.360601    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.360831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.360682    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.360831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.360713    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.360831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.360763    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.361330 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.361292    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-certs\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.361330 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.361309    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7198de3d-596e-4e43-a71c-d1dcf8deea73-istiod-ca-cert\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.361448 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.361302    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-credential-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.361583 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.361551    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-socket\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.361819 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.361794    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-data\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.363569 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.363546    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-envoy\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.363764 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.363746    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-podinfo\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.384610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.384541    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85nfd\" (UniqueName: \"kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-kube-api-access-85nfd\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.385100 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.385078    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-token\") pod \"e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") " pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.449469 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.449430    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:18.778056 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:54:18.778023    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7198de3d_596e_4e43_a71c_d1dcf8deea73.slice/crio-202d5a84ab844a93f62fec3200457dd717f37f3157fd80749815c71b27c1fac1 WatchSource:0}: Error finding container 202d5a84ab844a93f62fec3200457dd717f37f3157fd80749815c71b27c1fac1: Status 404 returned error can't find the container with id 202d5a84ab844a93f62fec3200457dd717f37f3157fd80749815c71b27c1fac1
Nov 02 07:54:18.781095 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.781049    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:18.781196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.781142    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:18.781196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.781187    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:18.783331 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:18.783302    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"]
Nov 02 07:54:19.046361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:19.046257    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" event={"ID":"7198de3d-596e-4e43-a71c-d1dcf8deea73","Type":"ContainerStarted","Data":"b32ef8138c6e5b6366eb55f8b6e228962a23d5b32c036ea0bf25a612a003f88c"}
Nov 02 07:54:19.046361 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:19.046327    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" event={"ID":"7198de3d-596e-4e43-a71c-d1dcf8deea73","Type":"ContainerStarted","Data":"202d5a84ab844a93f62fec3200457dd717f37f3157fd80749815c71b27c1fac1"}
Nov 02 07:54:19.097000 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:19.096942    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" podStartSLOduration=1.096923555 podStartE2EDuration="1.096923555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:19.091139476 +0000 UTC m=+1809.781466441" watchObservedRunningTime="2026-11-02 07:54:19.096923555 +0000 UTC m=+1809.787250517"
Nov 02 07:54:19.451285 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:19.451236    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:19.453233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:19.453203    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 02 07:54:19.453346 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:19.453249    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" podUID="7198de3d-596e-4e43-a71c-d1dcf8deea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 02 07:54:20.449993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:20.449950    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 02 07:54:20.450183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:20.450013    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" podUID="7198de3d-596e-4e43-a71c-d1dcf8deea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 02 07:54:21.451330 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:21.451296    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 02 07:54:21.451845 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:21.451351    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" podUID="7198de3d-596e-4e43-a71c-d1dcf8deea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 02 07:54:22.450025 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:22.449990    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 02 07:54:22.450238 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:22.450056    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" podUID="7198de3d-596e-4e43-a71c-d1dcf8deea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 02 07:54:23.450022 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:23.449985    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 02 07:54:23.450421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:23.450036    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" podUID="7198de3d-596e-4e43-a71c-d1dcf8deea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 02 07:54:24.450183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:24.450147    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 02 07:54:24.450591 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:24.450197    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" podUID="7198de3d-596e-4e43-a71c-d1dcf8deea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 02 07:54:25.453991 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:25.452322    4898 patch_prober.go:29] interesting pod/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 02 07:54:25.453991 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:25.452392    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" podUID="7198de3d-596e-4e43-a71c-d1dcf8deea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 02 07:54:26.995778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:26.995739    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:26.996295 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:26.996183    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:26.997747 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:26.997723    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:54:27.905546 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:27.905512    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"]
Nov 02 07:54:27.905783 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:27.905755    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64" podUID="3db07204-2fc3-4be1-9730-5f47f311a5ae" containerName="payload-processing" containerID="cri-o://07b7f99d3e245469cc299b9968327f07024831a5bc2d9ee915a33189d96e3f46" gracePeriod=30
Nov 02 07:54:28.229966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:28.228914    4898 generic.go:356] "Generic (PLEG): container finished" podID="3db07204-2fc3-4be1-9730-5f47f311a5ae" containerID="07b7f99d3e245469cc299b9968327f07024831a5bc2d9ee915a33189d96e3f46" exitCode=0
Nov 02 07:54:28.230474 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:28.230288    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64" event={"ID":"3db07204-2fc3-4be1-9730-5f47f311a5ae","Type":"ContainerDied","Data":"07b7f99d3e245469cc299b9968327f07024831a5bc2d9ee915a33189d96e3f46"}
Nov 02 07:54:28.465815 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:28.465782    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"
Nov 02 07:54:28.555814 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:28.555721    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db07204-2fc3-4be1-9730-5f47f311a5ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db07204-2fc3-4be1-9730-5f47f311a5ae-plugins-config-volume\") pod \"3db07204-2fc3-4be1-9730-5f47f311a5ae\" (UID: \"3db07204-2fc3-4be1-9730-5f47f311a5ae\") "
Nov 02 07:54:28.556041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:28.555959    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db07204-2fc3-4be1-9730-5f47f311a5ae-kube-api-access-lhtdg\" (UniqueName: \"kubernetes.io/projected/3db07204-2fc3-4be1-9730-5f47f311a5ae-kube-api-access-lhtdg\") pod \"3db07204-2fc3-4be1-9730-5f47f311a5ae\" (UID: \"3db07204-2fc3-4be1-9730-5f47f311a5ae\") "
Nov 02 07:54:28.556233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:28.556204    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db07204-2fc3-4be1-9730-5f47f311a5ae-plugins-config-volume" pod "3db07204-2fc3-4be1-9730-5f47f311a5ae" (UID: "3db07204-2fc3-4be1-9730-5f47f311a5ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:54:28.557055 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:28.557029    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db07204-2fc3-4be1-9730-5f47f311a5ae-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:54:28.586011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:28.585964    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db07204-2fc3-4be1-9730-5f47f311a5ae-kube-api-access-lhtdg" pod "3db07204-2fc3-4be1-9730-5f47f311a5ae" (UID: "3db07204-2fc3-4be1-9730-5f47f311a5ae"). InnerVolumeSpecName "kube-api-access-lhtdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:54:28.658828 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:28.658783    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhtdg\" (UniqueName: \"kubernetes.io/projected/3db07204-2fc3-4be1-9730-5f47f311a5ae-kube-api-access-lhtdg\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:54:29.238662 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:29.238577    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"
Nov 02 07:54:29.238662 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:29.238608    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64" event={"ID":"3db07204-2fc3-4be1-9730-5f47f311a5ae","Type":"ContainerDied","Data":"4d92d152b7cda3a6a5b1d62d6bcdb4c4763e6c48c65ee3d9675fa60ef31c928b"}
Nov 02 07:54:29.239188 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:29.238715    4898 scope.go:122] "RemoveContainer" containerID="07b7f99d3e245469cc299b9968327f07024831a5bc2d9ee915a33189d96e3f46"
Nov 02 07:54:29.297240 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:29.297201    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"]
Nov 02 07:54:29.312530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:29.312481    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-677726-78868ff7c8-sfp64"]
Nov 02 07:54:29.489755 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:29.489685    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db07204-2fc3-4be1-9730-5f47f311a5ae" path="/var/lib/kubelet/pods/3db07204-2fc3-4be1-9730-5f47f311a5ae/volumes"
Nov 02 07:54:33.201790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.201760    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"]
Nov 02 07:54:33.202489 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.202471    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3db07204-2fc3-4be1-9730-5f47f311a5ae" containerName="payload-processing"
Nov 02 07:54:33.202547 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.202490    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db07204-2fc3-4be1-9730-5f47f311a5ae" containerName="payload-processing"
Nov 02 07:54:33.202614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.202605    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db07204-2fc3-4be1-9730-5f47f311a5ae" containerName="payload-processing"
Nov 02 07:54:33.206352 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.206331    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.229037 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.229003    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c7a27f-openshift-default-dockercfg-4pqn9\""
Nov 02 07:54:33.250159 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.250114    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"]
Nov 02 07:54:33.297691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.297651    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-socket\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.297870 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.297844    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-podinfo\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.298019 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.298004    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-certs\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.298060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.298029    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd84a2fa-265f-442f-ad82-40be74f3c00c-istiod-ca-cert\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.298101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.298057    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-data\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.298101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.298078    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-credential-socket\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.298182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.298166    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-envoy\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.298219 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.298197    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdcj\" (UniqueName: \"kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-kube-api-access-qqdcj\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.298283 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.298272    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-token\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.460464    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-socket\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.460671    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-podinfo\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.460765    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-certs\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.460796    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd84a2fa-265f-442f-ad82-40be74f3c00c-istiod-ca-cert\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.460842    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-data\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.460871    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-credential-socket\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.460968    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-envoy\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.461002    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdcj\" (UniqueName: \"kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-kube-api-access-qqdcj\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.461060    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-token\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.461813    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-socket\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.463671    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-data\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.463951    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-certs\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.469966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.464669    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd84a2fa-265f-442f-ad82-40be74f3c00c-istiod-ca-cert\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.476166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.475058    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-credential-socket\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.495876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.495845    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-podinfo\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.501582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.501538    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-token\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.510023 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.509359    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-envoy\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.531921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.516965    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdcj\" (UniqueName: \"kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-kube-api-access-qqdcj\") pod \"e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") " pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.534081 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.533793    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:33.923792 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:54:33.923764    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd84a2fa_265f_442f_ad82_40be74f3c00c.slice/crio-6969ce6f92d8306f7476c9355c29e5177861f642b3007a94ccb55e4f0e6ad2ed WatchSource:0}: Error finding container 6969ce6f92d8306f7476c9355c29e5177861f642b3007a94ccb55e4f0e6ad2ed: Status 404 returned error can't find the container with id 6969ce6f92d8306f7476c9355c29e5177861f642b3007a94ccb55e4f0e6ad2ed
Nov 02 07:54:33.927002 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.926960    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:33.927110 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.927053    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:33.927110 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.927102    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:33.929475 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:33.929449    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"]
Nov 02 07:54:34.504289 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:34.503191    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" event={"ID":"fd84a2fa-265f-442f-ad82-40be74f3c00c","Type":"ContainerStarted","Data":"9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65"}
Nov 02 07:54:34.504289 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:34.503265    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" event={"ID":"fd84a2fa-265f-442f-ad82-40be74f3c00c","Type":"ContainerStarted","Data":"6969ce6f92d8306f7476c9355c29e5177861f642b3007a94ccb55e4f0e6ad2ed"}
Nov 02 07:54:34.536084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:34.536044    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:34.543143 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:34.543112    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:34.576907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:34.576848    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" podStartSLOduration=1.576832773 podStartE2EDuration="1.576832773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:34.561988951 +0000 UTC m=+1825.252315915" watchObservedRunningTime="2026-11-02 07:54:34.576832773 +0000 UTC m=+1825.267159813"
Nov 02 07:54:35.565267 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:35.549501    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:35.575533 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:35.575497    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:54:36.697438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.697395    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"]
Nov 02 07:54:36.708661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.708626    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"]
Nov 02 07:54:36.708966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.708789    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.724266 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.724234    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-3575b9-openshift-default-dockercfg-6pv8l\""
Nov 02 07:54:36.820362 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.820321    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-token\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.820596 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.820460    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-socket\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.820596 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.820563    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-credential-socket\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.820757 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.820609    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-certs\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.820757 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.820643    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-podinfo\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.820757 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.820740    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-data\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.821050 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.820768    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s92vn\" (UniqueName: \"kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-kube-api-access-s92vn\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.821050 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.820799    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istiod-ca-cert\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.821050 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.820864    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-envoy\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.922159 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.922118    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-certs\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.922381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.922182    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-podinfo\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.922381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.922235    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-data\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.922381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.922261    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s92vn\" (UniqueName: \"kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-kube-api-access-s92vn\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.922381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.922298    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istiod-ca-cert\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.922381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.922330    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-envoy\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.922733 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.922412    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-token\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.922733 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.922527    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-socket\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.922733 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.922637    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-data\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.922976 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.922641    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-credential-socket\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.923060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.923025    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-credential-socket\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.923173 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.923147    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istiod-ca-cert\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.923242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.923171    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-certs\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.923242 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.923217    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-socket\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.925250 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.925220    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-envoy\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.925979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.925952    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-podinfo\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:36.997693 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:36.997603    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s92vn\" (UniqueName: \"kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-kube-api-access-s92vn\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:37.008911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.008458    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-token\") pod \"e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") " pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:37.038190 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.038154    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:37.472129 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:54:37.472080    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529b9b31_ca08_40e9_815b_5f5afee7dd1d.slice/crio-e2ceaf19bb912160a38ea297562fbaf3dabb45b96ae19bc034c991576d423345 WatchSource:0}: Error finding container e2ceaf19bb912160a38ea297562fbaf3dabb45b96ae19bc034c991576d423345: Status 404 returned error can't find the container with id e2ceaf19bb912160a38ea297562fbaf3dabb45b96ae19bc034c991576d423345
Nov 02 07:54:37.475116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.475070    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:37.475230 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.475155    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"]
Nov 02 07:54:37.475230 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.475161    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:37.475325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.475278    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:37.560070 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.560041    4898 generic.go:356] "Generic (PLEG): container finished" podID="2c33009d-0656-4873-9be6-968f615c990d" containerID="897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b" exitCode=0
Nov 02 07:54:37.560211 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.560118    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" event={"ID":"2c33009d-0656-4873-9be6-968f615c990d","Type":"ContainerDied","Data":"897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b"}
Nov 02 07:54:37.560577 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.560556    4898 scope.go:122] "RemoveContainer" containerID="897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b"
Nov 02 07:54:37.562132 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.562086    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" event={"ID":"529b9b31-ca08-40e9-815b-5f5afee7dd1d","Type":"ContainerStarted","Data":"e2ceaf19bb912160a38ea297562fbaf3dabb45b96ae19bc034c991576d423345"}
Nov 02 07:54:37.562530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.562501    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:37.562686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.562587    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:37.562686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:37.562652    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:38.590673 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:38.590631    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" event={"ID":"2c33009d-0656-4873-9be6-968f615c990d","Type":"ContainerStarted","Data":"62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c"}
Nov 02 07:54:38.595222 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:38.595186    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" event={"ID":"529b9b31-ca08-40e9-815b-5f5afee7dd1d","Type":"ContainerStarted","Data":"a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef"}
Nov 02 07:54:38.722822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:38.722765    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" podStartSLOduration=2.722750101 podStartE2EDuration="2.722750101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:38.719387748 +0000 UTC m=+1829.409714713" watchObservedRunningTime="2026-11-02 07:54:38.722750101 +0000 UTC m=+1829.413077064"
Nov 02 07:54:39.017639 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:39.017610    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:54:39.018978 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:39.018946    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 07:54:39.019091 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:39.019030    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 07:54:39.039616 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:39.039582    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:39.045253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:39.045228    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:39.614024 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:39.613952    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:39.615015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:39.614985    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:54:39.802390 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:39.802355    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"]
Nov 02 07:54:39.806731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:39.806709    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"
Nov 02 07:54:39.826038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:39.826007    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"]
Nov 02 07:54:39.828628 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:39.828596    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c7a27f\""
Nov 02 07:54:39.828861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:39.828847    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c7a27f-dockercfg-xrwf7\""
Nov 02 07:54:40.093186 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:40.093147    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 07:54:40.093392 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:40.093209    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 07:54:40.228480 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:40.228428    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bztv9\" (UniqueName: \"kubernetes.io/projected/c899bc58-259d-42e1-a77e-8b3c26c7993e-kube-api-access-bztv9\") pod \"payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb\" (UID: \"c899bc58-259d-42e1-a77e-8b3c26c7993e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"
Nov 02 07:54:40.228702 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:40.228629    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c899bc58-259d-42e1-a77e-8b3c26c7993e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb\" (UID: \"c899bc58-259d-42e1-a77e-8b3c26c7993e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"
Nov 02 07:54:40.330031 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:40.329989    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c899bc58-259d-42e1-a77e-8b3c26c7993e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb\" (UID: \"c899bc58-259d-42e1-a77e-8b3c26c7993e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"
Nov 02 07:54:40.330251 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:40.330156    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bztv9\" (UniqueName: \"kubernetes.io/projected/c899bc58-259d-42e1-a77e-8b3c26c7993e-kube-api-access-bztv9\") pod \"payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb\" (UID: \"c899bc58-259d-42e1-a77e-8b3c26c7993e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"
Nov 02 07:54:40.330857 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:40.330827    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c899bc58-259d-42e1-a77e-8b3c26c7993e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb\" (UID: \"c899bc58-259d-42e1-a77e-8b3c26c7993e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"
Nov 02 07:54:40.353884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:40.353822    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bztv9\" (UniqueName: \"kubernetes.io/projected/c899bc58-259d-42e1-a77e-8b3c26c7993e-kube-api-access-bztv9\") pod \"payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb\" (UID: \"c899bc58-259d-42e1-a77e-8b3c26c7993e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"
Nov 02 07:54:40.422487 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:40.422450    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"
Nov 02 07:54:40.814241 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:40.812183    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"]
Nov 02 07:54:41.017644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:41.017604    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 07:54:41.017831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:41.017663    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 07:54:41.641627 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:41.641582    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb" event={"ID":"c899bc58-259d-42e1-a77e-8b3c26c7993e","Type":"ContainerStarted","Data":"5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd"}
Nov 02 07:54:41.641836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:41.641644    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb" event={"ID":"c899bc58-259d-42e1-a77e-8b3c26c7993e","Type":"ContainerStarted","Data":"f77c116661cf115da6dcc92d96dea28c69270faf287bc11b41f815ddff1d90af"}
Nov 02 07:54:41.641836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:41.641668    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"
Nov 02 07:54:41.709110 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:41.709052    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb" podStartSLOduration=2.709037113 podStartE2EDuration="2.709037113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:41.706794222 +0000 UTC m=+1832.397121181" watchObservedRunningTime="2026-11-02 07:54:41.709037113 +0000 UTC m=+1832.399364102"
Nov 02 07:54:42.017070 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:42.017033    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 07:54:42.017493 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:42.017091    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 07:54:43.017584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.017548    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 07:54:43.018106 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.017612    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 07:54:43.033334 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.033298    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"]
Nov 02 07:54:43.208223 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.205869    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2296317e-nmtzc"]
Nov 02 07:54:43.809215 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.809169    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6917528e-6e93-4266-b4e1-1e3a99f75195" path="/var/lib/kubelet/pods/6917528e-6e93-4266-b4e1-1e3a99f75195/volumes"
Nov 02 07:54:43.809749 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.809727    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"]
Nov 02 07:54:43.816178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.816146    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:54:43.816311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.816189    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"]
Nov 02 07:54:43.816311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.816247    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"
Nov 02 07:54:43.842999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.842971    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-3575b9-dockercfg-4c558\""
Nov 02 07:54:43.842999 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.842987    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-3575b9\""
Nov 02 07:54:43.909430 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.909385    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26wnf\" (UniqueName: \"kubernetes.io/projected/13a7ae74-c294-4de1-b499-a2e0a7318cfc-kube-api-access-26wnf\") pod \"payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj\" (UID: \"13a7ae74-c294-4de1-b499-a2e0a7318cfc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"
Nov 02 07:54:43.909721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:43.909691    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a7ae74-c294-4de1-b499-a2e0a7318cfc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj\" (UID: \"13a7ae74-c294-4de1-b499-a2e0a7318cfc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"
Nov 02 07:54:44.011278 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:44.011242    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26wnf\" (UniqueName: \"kubernetes.io/projected/13a7ae74-c294-4de1-b499-a2e0a7318cfc-kube-api-access-26wnf\") pod \"payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj\" (UID: \"13a7ae74-c294-4de1-b499-a2e0a7318cfc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"
Nov 02 07:54:44.011467 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:44.011307    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a7ae74-c294-4de1-b499-a2e0a7318cfc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj\" (UID: \"13a7ae74-c294-4de1-b499-a2e0a7318cfc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"
Nov 02 07:54:44.012118 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:44.012097    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a7ae74-c294-4de1-b499-a2e0a7318cfc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj\" (UID: \"13a7ae74-c294-4de1-b499-a2e0a7318cfc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"
Nov 02 07:54:44.017528 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:44.017501    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 07:54:44.017653 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:44.017563    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 07:54:44.036772 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:44.036744    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26wnf\" (UniqueName: \"kubernetes.io/projected/13a7ae74-c294-4de1-b499-a2e0a7318cfc-kube-api-access-26wnf\") pod \"payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj\" (UID: \"13a7ae74-c294-4de1-b499-a2e0a7318cfc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"
Nov 02 07:54:44.132180 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:44.132073    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"
Nov 02 07:54:44.678609 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:54:44.678557    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a7ae74_c294_4de1_b499_a2e0a7318cfc.slice/crio-70cc412417127bc73aa3d0dabab323710854590ee196e6eea69e3787632c475c WatchSource:0}: Error finding container 70cc412417127bc73aa3d0dabab323710854590ee196e6eea69e3787632c475c: Status 404 returned error can't find the container with id 70cc412417127bc73aa3d0dabab323710854590ee196e6eea69e3787632c475c
Nov 02 07:54:44.679579 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:44.679545    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"]
Nov 02 07:54:44.813911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:44.813471    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj" event={"ID":"13a7ae74-c294-4de1-b499-a2e0a7318cfc","Type":"ContainerStarted","Data":"ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d"}
Nov 02 07:54:44.813911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:44.813542    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj" event={"ID":"13a7ae74-c294-4de1-b499-a2e0a7318cfc","Type":"ContainerStarted","Data":"70cc412417127bc73aa3d0dabab323710854590ee196e6eea69e3787632c475c"}
Nov 02 07:54:44.813911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:44.813573    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"
Nov 02 07:54:44.860792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:44.860720    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj" podStartSLOduration=1.8607019390000001 podStartE2EDuration="1.860701939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:44.85673217 +0000 UTC m=+1835.547059162" watchObservedRunningTime="2026-11-02 07:54:44.860701939 +0000 UTC m=+1835.551028904"
Nov 02 07:54:45.094050 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:54:45.089089    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:54:45.150200 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:45.150169    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:54:45.155650 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:45.155620    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 07:54:52.655960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:52.655660    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"
Nov 02 07:54:54.308365 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.308314    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"]
Nov 02 07:54:54.315291 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.315262    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.340937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.339562    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-6929a4-openshift-default-dockercfg-dv4x2\""
Nov 02 07:54:54.340937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.340338    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"]
Nov 02 07:54:54.424993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.420910    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-certs\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.424993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.421021    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-token\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.424993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.421055    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-envoy\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.424993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.421087    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-podinfo\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.424993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.421148    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgtrx\" (UniqueName: \"kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-kube-api-access-qgtrx\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.424993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.421193    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb2e6c5-145c-4448-8404-1f676175dc7e-istiod-ca-cert\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.424993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.421262    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-socket\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.424993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.421315    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-data\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.424993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.421383    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-credential-socket\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.527914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.525974    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-data\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.527914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.526063    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-credential-socket\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.527914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.526246    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-certs\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.527914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.526315    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-token\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.527914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.526342    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-envoy\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.527914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.526372    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-podinfo\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.527914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.526436    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgtrx\" (UniqueName: \"kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-kube-api-access-qgtrx\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.527914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.526482    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb2e6c5-145c-4448-8404-1f676175dc7e-istiod-ca-cert\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.527914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.526554    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-socket\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.527914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.527118    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-socket\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.527914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.527621    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-data\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.527914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.527837    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-credential-socket\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.528738 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.528142    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-certs\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.546923 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.541769    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb2e6c5-145c-4448-8404-1f676175dc7e-istiod-ca-cert\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.547422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.547394    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-envoy\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.554176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.554112    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-podinfo\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.590584 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.590508    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-token\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.603946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.594085    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgtrx\" (UniqueName: \"kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-kube-api-access-qgtrx\") pod \"e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") " pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.640421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.640126    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:54.997006 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:54:54.995514    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb2e6c5_145c_4448_8404_1f676175dc7e.slice/crio-5a6be7c30b4d37e463b0dce6042a9c9b9ba7121ac6bc5b9d884d9a9700e322f6 WatchSource:0}: Error finding container 5a6be7c30b4d37e463b0dce6042a9c9b9ba7121ac6bc5b9d884d9a9700e322f6: Status 404 returned error can't find the container with id 5a6be7c30b4d37e463b0dce6042a9c9b9ba7121ac6bc5b9d884d9a9700e322f6
Nov 02 07:54:54.998518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.998473    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:54.998643 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.998573    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:54.998643 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:54.998623    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:54:55.000964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:55.000938    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"]
Nov 02 07:54:55.825381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:55.825352    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"
Nov 02 07:54:55.941580 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:55.941542    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" event={"ID":"4eb2e6c5-145c-4448-8404-1f676175dc7e","Type":"ContainerStarted","Data":"5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c"}
Nov 02 07:54:55.941767 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:55.941602    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" event={"ID":"4eb2e6c5-145c-4448-8404-1f676175dc7e","Type":"ContainerStarted","Data":"5a6be7c30b4d37e463b0dce6042a9c9b9ba7121ac6bc5b9d884d9a9700e322f6"}
Nov 02 07:54:55.990662 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:55.990603    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" podStartSLOduration=1.99058793 podStartE2EDuration="1.99058793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:54:55.987164968 +0000 UTC m=+1846.677491932" watchObservedRunningTime="2026-11-02 07:54:55.99058793 +0000 UTC m=+1846.680914893"
Nov 02 07:54:56.640605 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:56.640564    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:56.647032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:56.646994    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:56.950484 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:56.950453    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:56.951375 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:56.951351    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:54:59.055683 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:59.054400    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"]
Nov 02 07:54:59.069936 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:59.069900    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d54e3970-fsvnd"]
Nov 02 07:54:59.467260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:54:59.467227    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644352e9-29ba-478b-b055-f8405b740fd6" path="/var/lib/kubelet/pods/644352e9-29ba-478b-b055-f8405b740fd6/volumes"
Nov 02 07:55:12.034721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:12.034685    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"]
Nov 02 07:55:12.046703 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:12.046674    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab8356e1-64f5b"]
Nov 02 07:55:13.467233 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:13.467185    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc2c837d-6c5d-4f3c-ad59-7503f94c997e" path="/var/lib/kubelet/pods/bc2c837d-6c5d-4f3c-ad59-7503f94c997e/volumes"
Nov 02 07:55:14.989924 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:55:14.989867    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:55:25.035440 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:25.035399    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"]
Nov 02 07:55:25.046905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:25.046865    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9c3b2d-c8s9p"]
Nov 02 07:55:25.466254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:25.466216    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71db5b5-4f54-43ab-ba8d-05e114104d7d" path="/var/lib/kubelet/pods/d71db5b5-4f54-43ab-ba8d-05e114104d7d/volumes"
Nov 02 07:55:29.034654 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:29.034613    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"]
Nov 02 07:55:29.046510 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:29.046480    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c590519-4cvjz"]
Nov 02 07:55:29.557753 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:29.557705    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f" path="/var/lib/kubelet/pods/ca3e3f0a-c2f3-4394-959c-3d8cc3ecb77f/volumes"
Nov 02 07:55:34.718915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.676949    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"]
Nov 02 07:55:34.742535 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.725511    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:34.766015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.760430    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-680c01-openshift-default-dockercfg-flpwg\""
Nov 02 07:55:34.798921 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.790399    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"]
Nov 02 07:55:34.903270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.889145    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef52159-5187-4636-8278-b7405428e66d-istio-podinfo\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:34.903270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.889233    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-data\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:34.903270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.889284    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-certs\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:34.903270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.889311    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef52159-5187-4636-8278-b7405428e66d-istiod-ca-cert\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:34.903270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.889438    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9wz8\" (UniqueName: \"kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-kube-api-access-d9wz8\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:34.903270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.889496    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-socket\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:34.903270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.889532    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-envoy\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:34.903270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.889614    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-credential-socket\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:34.903270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:34.889748    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-istio-token\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.054875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.054785    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-socket\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.059003 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.058970    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-envoy\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.059249 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.059227    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-credential-socket\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.063070 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.063043    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-istio-token\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.063274 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.063255    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef52159-5187-4636-8278-b7405428e66d-istio-podinfo\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.063445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.063427    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-data\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.063596 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.063578    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-certs\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.069979 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.069950    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef52159-5187-4636-8278-b7405428e66d-istiod-ca-cert\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.070270 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.070248    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9wz8\" (UniqueName: \"kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-kube-api-access-d9wz8\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.071594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.071571    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-socket\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.099911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.089276    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-data\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.100504 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.100467    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-credential-socket\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.108905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.106979    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9wz8\" (UniqueName: \"kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-kube-api-access-d9wz8\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.108905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.107652    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef52159-5187-4636-8278-b7405428e66d-istiod-ca-cert\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.108905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.107941    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-certs\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.123087 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.123060    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-envoy\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.123308 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.123203    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef52159-5187-4636-8278-b7405428e66d-istio-podinfo\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.128534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.128506    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-istio-token\") pod \"e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") " pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:35.349048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:35.348943    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:36.248744 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:55:36.248710    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef52159_5187_4636_8278_b7405428e66d.slice/crio-79ef5444e5adbbb3ba6b1b33ff2363ad182774c9c083979d0044829de93537bf WatchSource:0}: Error finding container 79ef5444e5adbbb3ba6b1b33ff2363ad182774c9c083979d0044829de93537bf: Status 404 returned error can't find the container with id 79ef5444e5adbbb3ba6b1b33ff2363ad182774c9c083979d0044829de93537bf
Nov 02 07:55:36.251494 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:36.251469    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"]
Nov 02 07:55:36.251627 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:36.251506    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:55:36.251627 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:36.251570    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:55:36.251627 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:36.251600    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:55:36.638673 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:36.638564    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" event={"ID":"bef52159-5187-4636-8278-b7405428e66d","Type":"ContainerStarted","Data":"6b4c6504f7a6dfbfc567b5e5cd3b7657013291a20e9c02f6fb6375c754983200"}
Nov 02 07:55:36.638934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:36.638912    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" event={"ID":"bef52159-5187-4636-8278-b7405428e66d","Type":"ContainerStarted","Data":"79ef5444e5adbbb3ba6b1b33ff2363ad182774c9c083979d0044829de93537bf"}
Nov 02 07:55:36.727813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:36.727748    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" podStartSLOduration=2.727727293 podStartE2EDuration="2.727727293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:55:36.718248015 +0000 UTC m=+1887.408575004" watchObservedRunningTime="2026-11-02 07:55:36.727727293 +0000 UTC m=+1887.418054258"
Nov 02 07:55:37.349378 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:37.349333    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:37.371206 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:37.371174    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:37.649027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:37.648293    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:37.649421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:37.649387    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:55:38.041037 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:38.040990    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"]
Nov 02 07:55:38.046686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:38.046659    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1936d576-bggck"]
Nov 02 07:55:39.469072 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:39.469029    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0eb42b-9a10-49a3-8ad6-97a7bff68929" path="/var/lib/kubelet/pods/db0eb42b-9a10-49a3-8ad6-97a7bff68929/volumes"
Nov 02 07:55:40.039209 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:40.039166    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"]
Nov 02 07:55:40.050840 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:40.050804    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd704e4-zdsts"]
Nov 02 07:55:40.657541 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:40.656025    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:55:40.657541 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:40.656096    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:55:40.971066 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:40.970968    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"]
Nov 02 07:55:40.977745 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:40.977708    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"
Nov 02 07:55:40.997548 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:40.997494    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"]
Nov 02 07:55:41.002733 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.002696    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-680c01-dockercfg-4bsnv\""
Nov 02 07:55:41.002937 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.002773    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-680c01\""
Nov 02 07:55:41.070938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.068843    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8bw\" (UniqueName: \"kubernetes.io/projected/e28bd504-3ad1-4203-83d9-4d5156ac6390-kube-api-access-ws8bw\") pod \"payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm\" (UID: \"e28bd504-3ad1-4203-83d9-4d5156ac6390\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"
Nov 02 07:55:41.070938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.069003    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28bd504-3ad1-4203-83d9-4d5156ac6390-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm\" (UID: \"e28bd504-3ad1-4203-83d9-4d5156ac6390\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"
Nov 02 07:55:41.077259 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.077223    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"]
Nov 02 07:55:41.092229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.092195    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"
Nov 02 07:55:41.145130 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.145090    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"]
Nov 02 07:55:41.210914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.183959    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c82e96-5e12-43ff-aaeb-832e929263e4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f\" (UID: \"d7c82e96-5e12-43ff-aaeb-832e929263e4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"
Nov 02 07:55:41.224691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.211389    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8bw\" (UniqueName: \"kubernetes.io/projected/e28bd504-3ad1-4203-83d9-4d5156ac6390-kube-api-access-ws8bw\") pod \"payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm\" (UID: \"e28bd504-3ad1-4203-83d9-4d5156ac6390\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"
Nov 02 07:55:41.238764 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.225240    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28bd504-3ad1-4203-83d9-4d5156ac6390-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm\" (UID: \"e28bd504-3ad1-4203-83d9-4d5156ac6390\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"
Nov 02 07:55:41.239214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.239182    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w499s\" (UniqueName: \"kubernetes.io/projected/d7c82e96-5e12-43ff-aaeb-832e929263e4-kube-api-access-w499s\") pod \"payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f\" (UID: \"d7c82e96-5e12-43ff-aaeb-832e929263e4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"
Nov 02 07:55:41.239508 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.226580    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28bd504-3ad1-4203-83d9-4d5156ac6390-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm\" (UID: \"e28bd504-3ad1-4203-83d9-4d5156ac6390\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"
Nov 02 07:55:41.264494 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.243860    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8bw\" (UniqueName: \"kubernetes.io/projected/e28bd504-3ad1-4203-83d9-4d5156ac6390-kube-api-access-ws8bw\") pod \"payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm\" (UID: \"e28bd504-3ad1-4203-83d9-4d5156ac6390\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"
Nov 02 07:55:41.327617 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.324903    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"
Nov 02 07:55:41.349426 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.347870    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c82e96-5e12-43ff-aaeb-832e929263e4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f\" (UID: \"d7c82e96-5e12-43ff-aaeb-832e929263e4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"
Nov 02 07:55:41.350532 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.349937    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w499s\" (UniqueName: \"kubernetes.io/projected/d7c82e96-5e12-43ff-aaeb-832e929263e4-kube-api-access-w499s\") pod \"payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f\" (UID: \"d7c82e96-5e12-43ff-aaeb-832e929263e4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"
Nov 02 07:55:41.358364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.358305    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c82e96-5e12-43ff-aaeb-832e929263e4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f\" (UID: \"d7c82e96-5e12-43ff-aaeb-832e929263e4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"
Nov 02 07:55:41.381325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.381290    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w499s\" (UniqueName: \"kubernetes.io/projected/d7c82e96-5e12-43ff-aaeb-832e929263e4-kube-api-access-w499s\") pod \"payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f\" (UID: \"d7c82e96-5e12-43ff-aaeb-832e929263e4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"
Nov 02 07:55:41.382037 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.381832    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"
Nov 02 07:55:41.539215 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.539179    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbbceb31-c5f2-4b7a-8917-adf105c31df0" path="/var/lib/kubelet/pods/fbbceb31-c5f2-4b7a-8917-adf105c31df0/volumes"
Nov 02 07:55:41.723602 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:55:41.723561    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28bd504_3ad1_4203_83d9_4d5156ac6390.slice/crio-4de6e6ef73aaf465925823405599a0d355b3102c687a877f4377d62b4c8eb58c WatchSource:0}: Error finding container 4de6e6ef73aaf465925823405599a0d355b3102c687a877f4377d62b4c8eb58c: Status 404 returned error can't find the container with id 4de6e6ef73aaf465925823405599a0d355b3102c687a877f4377d62b4c8eb58c
Nov 02 07:55:41.724640 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.724612    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"]
Nov 02 07:55:41.773750 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:55:41.773722    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c82e96_5e12_43ff_aaeb_832e929263e4.slice/crio-4a8cd352c6e6cc43c5bd0863ee1244b191d82e0ccbbf2b6ddef0ab766e103943 WatchSource:0}: Error finding container 4a8cd352c6e6cc43c5bd0863ee1244b191d82e0ccbbf2b6ddef0ab766e103943: Status 404 returned error can't find the container with id 4a8cd352c6e6cc43c5bd0863ee1244b191d82e0ccbbf2b6ddef0ab766e103943
Nov 02 07:55:41.780519 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:41.780492    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"]
Nov 02 07:55:42.734304 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:42.734251    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm" event={"ID":"e28bd504-3ad1-4203-83d9-4d5156ac6390","Type":"ContainerStarted","Data":"5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed"}
Nov 02 07:55:42.735223 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:42.734321    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm" event={"ID":"e28bd504-3ad1-4203-83d9-4d5156ac6390","Type":"ContainerStarted","Data":"4de6e6ef73aaf465925823405599a0d355b3102c687a877f4377d62b4c8eb58c"}
Nov 02 07:55:42.735223 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:42.734377    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"
Nov 02 07:55:42.736661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:42.736625    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f" event={"ID":"d7c82e96-5e12-43ff-aaeb-832e929263e4","Type":"ContainerStarted","Data":"92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee"}
Nov 02 07:55:42.737147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:42.736690    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f" event={"ID":"d7c82e96-5e12-43ff-aaeb-832e929263e4","Type":"ContainerStarted","Data":"4a8cd352c6e6cc43c5bd0863ee1244b191d82e0ccbbf2b6ddef0ab766e103943"}
Nov 02 07:55:42.737147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:42.736722    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"
Nov 02 07:55:42.781541 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:42.781470    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm" podStartSLOduration=2.7814515650000002 podStartE2EDuration="2.781451565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:55:42.77500079 +0000 UTC m=+1893.465327780" watchObservedRunningTime="2026-11-02 07:55:42.781451565 +0000 UTC m=+1893.471778531"
Nov 02 07:55:42.817761 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:42.817702    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f" podStartSLOduration=1.817689804 podStartE2EDuration="1.817689804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:55:42.815290655 +0000 UTC m=+1893.505617642" watchObservedRunningTime="2026-11-02 07:55:42.817689804 +0000 UTC m=+1893.508016767"
Nov 02 07:55:44.757732 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:44.757697    4898 generic.go:356] "Generic (PLEG): container finished" podID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerID="a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef" exitCode=0
Nov 02 07:55:44.758197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:44.757741    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" event={"ID":"529b9b31-ca08-40e9-815b-5f5afee7dd1d","Type":"ContainerDied","Data":"a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef"}
Nov 02 07:55:44.758338 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:44.758319    4898 scope.go:122] "RemoveContainer" containerID="a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef"
Nov 02 07:55:44.760487 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:44.760450    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:55:44.760618 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:44.760549    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:55:44.760618 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:44.760596    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:55:45.004182 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:55:45.004077    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:55:45.768612 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:45.768570    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" event={"ID":"529b9b31-ca08-40e9-815b-5f5afee7dd1d","Type":"ContainerStarted","Data":"cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34"}
Nov 02 07:55:45.770572 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:45.770547    4898 generic.go:356] "Generic (PLEG): container finished" podID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerID="9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65" exitCode=0
Nov 02 07:55:45.770733 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:45.770616    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" event={"ID":"fd84a2fa-265f-442f-ad82-40be74f3c00c","Type":"ContainerDied","Data":"9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65"}
Nov 02 07:55:45.771018 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:45.771003    4898 scope.go:122] "RemoveContainer" containerID="9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65"
Nov 02 07:55:45.772868 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:45.772832    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:55:45.772986 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:45.772940    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:55:45.773061 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:45.773009    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:55:46.039128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:46.039045    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:55:46.039310 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:46.039285    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 07:55:46.039368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:46.039346    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 07:55:46.812915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:46.811005    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" event={"ID":"fd84a2fa-265f-442f-ad82-40be74f3c00c","Type":"ContainerStarted","Data":"3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5"}
Nov 02 07:55:47.045915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:47.039013    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 07:55:47.045915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:47.039076    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 07:55:47.539904 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:47.539861    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:55:47.541707 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:47.541675    4898 patch_prober.go:29] interesting pod/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 07:55:47.541852 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:47.541728    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 07:55:48.039372 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:48.039341    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 07:55:48.039864 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:48.039836    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 07:55:48.535445 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:48.535410    4898 patch_prober.go:29] interesting pod/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 07:55:48.535633 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:48.535455    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 07:55:49.039433 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:49.039397    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 07:55:49.039844 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:49.039455    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 07:55:49.537104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:49.536435    4898 patch_prober.go:29] interesting pod/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 07:55:49.537104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:49.536488    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 07:55:50.039476 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:50.039435    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 07:55:50.040030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:50.039503    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 07:55:50.538081 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:50.538044    4898 patch_prober.go:29] interesting pod/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 07:55:50.538262 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:50.538103    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 07:55:50.549306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:50.549267    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:55:51.039363 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:51.039322    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 02 07:55:51.039616 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:51.039399    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 02 07:55:51.535925 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:51.535868    4898 patch_prober.go:29] interesting pod/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 02 07:55:51.536153 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:51.535942    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 02 07:55:52.043667 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:52.043634    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:55:52.044149 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:52.044133    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:55:52.044808 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:52.044789    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 07:55:52.659934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:52.659902    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:55:52.677907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:52.673652    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:55:53.749871 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:53.749838    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"
Nov 02 07:55:53.750518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:53.750497    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"
Nov 02 07:55:54.041059 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:54.040971    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"]
Nov 02 07:55:54.047162 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:54.047123    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"]
Nov 02 07:55:54.052977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:54.052944    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88591333-4k9bc"]
Nov 02 07:55:54.058931 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:54.058905    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5069c276-tjl44"]
Nov 02 07:55:55.487401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:55.487360    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92ad256-0ca7-4282-9991-a67bcc91ca18" path="/var/lib/kubelet/pods/b92ad256-0ca7-4282-9991-a67bcc91ca18/volumes"
Nov 02 07:55:55.488131 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:55:55.488105    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0729dd4-778e-4dcd-af53-be08c31754fa" path="/var/lib/kubelet/pods/d0729dd4-778e-4dcd-af53-be08c31754fa/volumes"
Nov 02 07:56:02.033113 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:02.033076    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"]
Nov 02 07:56:02.053615 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:02.053570    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7e782b-wk475"]
Nov 02 07:56:03.470180 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:03.470139    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d25f037-09e4-49c4-a749-0e9715806f73" path="/var/lib/kubelet/pods/7d25f037-09e4-49c4-a749-0e9715806f73/volumes"
Nov 02 07:56:04.159936 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:04.159857    4898 generic.go:356] "Generic (PLEG): container finished" podID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerID="5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c" exitCode=0
Nov 02 07:56:04.160111 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:04.159936    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" event={"ID":"4eb2e6c5-145c-4448-8404-1f676175dc7e","Type":"ContainerDied","Data":"5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c"}
Nov 02 07:56:04.160381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:04.160367    4898 scope.go:122] "RemoveContainer" containerID="5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c"
Nov 02 07:56:04.163435 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:04.163391    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:56:04.163548 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:04.163487    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:56:04.163548 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:04.163536    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:56:05.193260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:05.193227    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" event={"ID":"4eb2e6c5-145c-4448-8404-1f676175dc7e","Type":"ContainerStarted","Data":"5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add"}
Nov 02 07:56:05.640380 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:05.640289    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:56:05.640562 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:05.640377    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 07:56:05.640562 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:05.640445    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 07:56:06.640476 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:06.640439    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 07:56:06.640915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:06.640526    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 07:56:07.640561 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:07.640525    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 07:56:07.641023 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:07.640605    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 07:56:08.640602 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:08.640563    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 07:56:08.641166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:08.640626    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 07:56:09.640422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:09.640390    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 07:56:09.640636 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:09.640448    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 07:56:10.033005 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:10.032972    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"]
Nov 02 07:56:10.044692 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:10.044666    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc37371-rl2r5"]
Nov 02 07:56:10.646550 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:10.646519    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:56:10.647104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:10.647082    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:56:10.647751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:10.647728    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 07:56:11.459437 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:11.459406    4898 scope.go:122] "RemoveContainer" containerID="8f79e031a73df28c9feb5a30c1df185f512bd6162abe3c31f5383c7007d528ad"
Nov 02 07:56:11.472368 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:11.472334    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f95feb-7b41-431b-bdf4-ed81c4a58119" path="/var/lib/kubelet/pods/99f95feb-7b41-431b-bdf4-ed81c4a58119/volumes"
Nov 02 07:56:11.474488 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:11.474465    4898 scope.go:122] "RemoveContainer" containerID="18ebb6cf3b5a36a34a423b0ea332f4a87d91b1876906716818aec2c37a0546ce"
Nov 02 07:56:11.487741 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:11.487720    4898 scope.go:122] "RemoveContainer" containerID="ca2365921dd3e74c84a27425d075c201fa6dc25c99a5e0f19e3a8c39db2a328e"
Nov 02 07:56:14.990292 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:56:14.990255    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:56:19.031135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:19.031093    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"]
Nov 02 07:56:19.042787 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:19.042753    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cadd308b-mvvxn"]
Nov 02 07:56:19.466273 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:19.466235    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e18c66-1de9-4087-9a95-91af68c032b3" path="/var/lib/kubelet/pods/42e18c66-1de9-4087-9a95-91af68c032b3/volumes"
Nov 02 07:56:31.717034 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.716943    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"]
Nov 02 07:56:31.725356 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.725329    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.741532 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.741506    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-3575b9\"/\"kube-root-ca.crt\""
Nov 02 07:56:31.742337 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.741724    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-3575b9\"/\"openshift-service-ca.crt\""
Nov 02 07:56:31.752347 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.752323    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"]
Nov 02 07:56:31.759911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.755165    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-3575b9\"/\"default-dockercfg-k6qtb\""
Nov 02 07:56:31.759911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.755285    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-3575b9\"/\"auth-test-model-3575b9-kserve-self-signed-certs\""
Nov 02 07:56:31.838522 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.838490    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-home\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.838688 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.838561    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5f52e9-580d-4f4d-b272-463a3e22b277-tls-certs\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.838688 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.838639    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-dshm\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.838765 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.838709    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-model-cache\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.838947 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.838929    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-tmp-dir\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.838998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.838980    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txx2\" (UniqueName: \"kubernetes.io/projected/bf5f52e9-580d-4f4d-b272-463a3e22b277-kube-api-access-2txx2\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.963823 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.963775    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-home\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.964201 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.964173    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5f52e9-580d-4f4d-b272-463a3e22b277-tls-certs\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.964451 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.964430    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-dshm\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.964714 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.964685    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-model-cache\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.965208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.965184    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-tmp-dir\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.965396 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.965378    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txx2\" (UniqueName: \"kubernetes.io/projected/bf5f52e9-580d-4f4d-b272-463a3e22b277-kube-api-access-2txx2\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.983438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.983362    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-home\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.985038 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.985012    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-model-cache\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.985381 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.985364    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-tmp-dir\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.996198 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.996167    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-dshm\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:31.997778 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:31.997752    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txx2\" (UniqueName: \"kubernetes.io/projected/bf5f52e9-580d-4f4d-b272-463a3e22b277-kube-api-access-2txx2\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:32.024946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:32.024877    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5f52e9-580d-4f4d-b272-463a3e22b277-tls-certs\") pod \"auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") " pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:32.044804 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:32.044768    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:32.528558 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:32.528511    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"]
Nov 02 07:56:32.626258 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:32.626228    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj" event={"ID":"bf5f52e9-580d-4f4d-b272-463a3e22b277","Type":"ContainerStarted","Data":"4571ca4d5d541bc374248d2e3f5fa6def2e7bcd5251507916cc04275bdd6cdd2"}
Nov 02 07:56:33.636723 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:33.636686    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj" event={"ID":"bf5f52e9-580d-4f4d-b272-463a3e22b277","Type":"ContainerStarted","Data":"398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c"}
Nov 02 07:56:33.683355 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:33.683065    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj" podStartSLOduration=2.683045302 podStartE2EDuration="2.683045302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:56:33.678714623 +0000 UTC m=+1944.369041614" watchObservedRunningTime="2026-11-02 07:56:33.683045302 +0000 UTC m=+1944.373372272"
Nov 02 07:56:34.045519 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:34.045482    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:36.046487 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:36.045374    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"]
Nov 02 07:56:36.080174 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:36.080076    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02a0fc2d-slmw5"]
Nov 02 07:56:37.574370 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:37.574330    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c1fe234-c577-4dbc-a3a8-2f66532e9be7" path="/var/lib/kubelet/pods/1c1fe234-c577-4dbc-a3a8-2f66532e9be7/volumes"
Nov 02 07:56:38.690071 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:38.690025    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 07:56:38.690521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:38.690089    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:56:39.722214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:39.722126    4898 generic.go:356] "Generic (PLEG): container finished" podID="bef52159-5187-4636-8278-b7405428e66d" containerID="6b4c6504f7a6dfbfc567b5e5cd3b7657013291a20e9c02f6fb6375c754983200" exitCode=0
Nov 02 07:56:39.722214 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:39.722184    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" event={"ID":"bef52159-5187-4636-8278-b7405428e66d","Type":"ContainerDied","Data":"6b4c6504f7a6dfbfc567b5e5cd3b7657013291a20e9c02f6fb6375c754983200"}
Nov 02 07:56:39.722743 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:39.722608    4898 scope.go:122] "RemoveContainer" containerID="6b4c6504f7a6dfbfc567b5e5cd3b7657013291a20e9c02f6fb6375c754983200"
Nov 02 07:56:39.724193 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:39.724163    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:56:39.724263 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:39.724235    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:56:39.724301 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:39.724269    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:56:40.732511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:40.732474    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" event={"ID":"bef52159-5187-4636-8278-b7405428e66d","Type":"ContainerStarted","Data":"b8618d5f939b66fa9a2a091bb922ca499a1df4c8b141baa0d1eefb934fa01f3f"}
Nov 02 07:56:41.351662 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:41.351459    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:56:41.351662 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:41.351609    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 07:56:41.352260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:41.351675    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 07:56:42.046413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:42.046379    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:42.089831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:42.089794    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:42.349687 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:42.349606    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 07:56:42.349837 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:42.349680    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 07:56:42.790367 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:42.790330    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 07:56:43.350176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:43.350137    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 07:56:43.350619 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:43.350205    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 07:56:44.349574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:44.349539    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 07:56:44.349761 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:44.349604    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 07:56:45.043809 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:56:45.040714    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:56:45.349945 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:45.349847    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 07:56:45.350093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:45.349936    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 07:56:46.349824 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:46.349783    4898 patch_prober.go:29] interesting pod/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 07:56:46.350265 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:46.349870    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 07:56:47.354267 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:47.354235    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:56:47.354754 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:47.354727    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:56:47.355528 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:47.355505    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:56:48.047476 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:48.047438    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"]
Nov 02 07:56:49.130982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:49.130942    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"]
Nov 02 07:56:49.158792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:49.158741    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a32ed4f-fxhnk"]
Nov 02 07:56:49.479251 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:49.479209    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1678db81-b5f4-4bfe-9fce-316f7104bd3a" path="/var/lib/kubelet/pods/1678db81-b5f4-4bfe-9fce-316f7104bd3a/volumes"
Nov 02 07:56:51.034421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:51.034386    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"]
Nov 02 07:56:51.045898 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:51.045856    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b33391f3-hqn2n"]
Nov 02 07:56:51.466695 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:51.466660    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a7e857-1289-4d52-8b85-c7af88c061ed" path="/var/lib/kubelet/pods/05a7e857-1289-4d52-8b85-c7af88c061ed/volumes"
Nov 02 07:56:53.047905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.047840    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb" podUID="c899bc58-259d-42e1-a77e-8b3c26c7993e" containerName="payload-pre-processing" containerID="cri-o://5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd" gracePeriod=25
Nov 02 07:56:53.354148 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.354116    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"
Nov 02 07:56:53.494178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.494146    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c899bc58-259d-42e1-a77e-8b3c26c7993e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c899bc58-259d-42e1-a77e-8b3c26c7993e-plugins-config-volume\") pod \"c899bc58-259d-42e1-a77e-8b3c26c7993e\" (UID: \"c899bc58-259d-42e1-a77e-8b3c26c7993e\") "
Nov 02 07:56:53.494385 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.494220    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c899bc58-259d-42e1-a77e-8b3c26c7993e-kube-api-access-bztv9\" (UniqueName: \"kubernetes.io/projected/c899bc58-259d-42e1-a77e-8b3c26c7993e-kube-api-access-bztv9\") pod \"c899bc58-259d-42e1-a77e-8b3c26c7993e\" (UID: \"c899bc58-259d-42e1-a77e-8b3c26c7993e\") "
Nov 02 07:56:53.494629 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.494597    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c899bc58-259d-42e1-a77e-8b3c26c7993e-plugins-config-volume" pod "c899bc58-259d-42e1-a77e-8b3c26c7993e" (UID: "c899bc58-259d-42e1-a77e-8b3c26c7993e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:56:53.494895 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.494871    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c899bc58-259d-42e1-a77e-8b3c26c7993e-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:56:53.496527 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.496504    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c899bc58-259d-42e1-a77e-8b3c26c7993e-kube-api-access-bztv9" pod "c899bc58-259d-42e1-a77e-8b3c26c7993e" (UID: "c899bc58-259d-42e1-a77e-8b3c26c7993e"). InnerVolumeSpecName "kube-api-access-bztv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:56:53.596251 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.596153    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bztv9\" (UniqueName: \"kubernetes.io/projected/c899bc58-259d-42e1-a77e-8b3c26c7993e-kube-api-access-bztv9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:56:53.938589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.938541    4898 generic.go:356] "Generic (PLEG): container finished" podID="c899bc58-259d-42e1-a77e-8b3c26c7993e" containerID="5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd" exitCode=0
Nov 02 07:56:53.938589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.938592    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"
Nov 02 07:56:53.938831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.938611    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb" event={"ID":"c899bc58-259d-42e1-a77e-8b3c26c7993e","Type":"ContainerDied","Data":"5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd"}
Nov 02 07:56:53.938831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.938674    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb" event={"ID":"c899bc58-259d-42e1-a77e-8b3c26c7993e","Type":"ContainerDied","Data":"f77c116661cf115da6dcc92d96dea28c69270faf287bc11b41f815ddff1d90af"}
Nov 02 07:56:53.938831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.938699    4898 scope.go:122] "RemoveContainer" containerID="5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd"
Nov 02 07:56:53.957376 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.957354    4898 scope.go:122] "RemoveContainer" containerID="5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd"
Nov 02 07:56:53.957724 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:56:53.957699    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd\": container with ID starting with 5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd not found: ID does not exist" containerID="5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd"
Nov 02 07:56:53.957842 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.957729    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd"} err="failed to get container status \"5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd\": rpc error: code = NotFound desc = could not find container \"5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd\": container with ID starting with 5ba8c61ed8cae17f0d6cba69b32bbd09e90ec1aec13536c0d2ef5c29693516bd not found: ID does not exist"
Nov 02 07:56:53.994573 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:53.994543    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"]
Nov 02 07:56:54.006835 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:54.006805    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c7a27f-867c5cd867-g4wrb"]
Nov 02 07:56:55.470841 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:55.470792    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c899bc58-259d-42e1-a77e-8b3c26c7993e" path="/var/lib/kubelet/pods/c899bc58-259d-42e1-a77e-8b3c26c7993e/volumes"
Nov 02 07:56:56.192199 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:56.192164    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"]
Nov 02 07:56:56.192734 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:56:56.192699    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerName="istio-proxy" containerID="cri-o://3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5" gracePeriod=30
Nov 02 07:57:01.517083 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.517052    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:57:01.594565 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.594526    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-kube-api-access-qqdcj\" (UniqueName: \"kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-kube-api-access-qqdcj\") pod \"fd84a2fa-265f-442f-ad82-40be74f3c00c\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") "
Nov 02 07:57:01.594791 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.594616    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-podinfo\") pod \"fd84a2fa-265f-442f-ad82-40be74f3c00c\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") "
Nov 02 07:57:01.594791 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.594710    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-credential-socket\") pod \"fd84a2fa-265f-442f-ad82-40be74f3c00c\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") "
Nov 02 07:57:01.594791 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.594784    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-data\") pod \"fd84a2fa-265f-442f-ad82-40be74f3c00c\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") "
Nov 02 07:57:01.595015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.594861    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-envoy\") pod \"fd84a2fa-265f-442f-ad82-40be74f3c00c\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") "
Nov 02 07:57:01.595538 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.595243    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-credential-socket" pod "fd84a2fa-265f-442f-ad82-40be74f3c00c" (UID: "fd84a2fa-265f-442f-ad82-40be74f3c00c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:57:01.595669 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.595555    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-data" pod "fd84a2fa-265f-442f-ad82-40be74f3c00c" (UID: "fd84a2fa-265f-442f-ad82-40be74f3c00c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:57:01.595739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.595676    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-token\" (UniqueName: \"kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-token\") pod \"fd84a2fa-265f-442f-ad82-40be74f3c00c\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") "
Nov 02 07:57:01.595799 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.595742    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd84a2fa-265f-442f-ad82-40be74f3c00c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd84a2fa-265f-442f-ad82-40be74f3c00c-istiod-ca-cert\") pod \"fd84a2fa-265f-442f-ad82-40be74f3c00c\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") "
Nov 02 07:57:01.595799 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.595773    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-certs\") pod \"fd84a2fa-265f-442f-ad82-40be74f3c00c\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") "
Nov 02 07:57:01.595930 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.595809    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-socket\") pod \"fd84a2fa-265f-442f-ad82-40be74f3c00c\" (UID: \"fd84a2fa-265f-442f-ad82-40be74f3c00c\") "
Nov 02 07:57:01.596412 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.596383    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-certs" pod "fd84a2fa-265f-442f-ad82-40be74f3c00c" (UID: "fd84a2fa-265f-442f-ad82-40be74f3c00c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:57:01.596582 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.596545    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-socket" pod "fd84a2fa-265f-442f-ad82-40be74f3c00c" (UID: "fd84a2fa-265f-442f-ad82-40be74f3c00c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:57:01.597012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.596988    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd84a2fa-265f-442f-ad82-40be74f3c00c-istiod-ca-cert" pod "fd84a2fa-265f-442f-ad82-40be74f3c00c" (UID: "fd84a2fa-265f-442f-ad82-40be74f3c00c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:57:01.597754 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.597730    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:01.597927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.597873    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd84a2fa-265f-442f-ad82-40be74f3c00c-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:01.598119 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.598102    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:01.598230 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.598218    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:01.598329 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.598315    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:01.598993 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.598952    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-envoy" pod "fd84a2fa-265f-442f-ad82-40be74f3c00c" (UID: "fd84a2fa-265f-442f-ad82-40be74f3c00c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:57:01.599420 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.599384    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-kube-api-access-qqdcj" pod "fd84a2fa-265f-442f-ad82-40be74f3c00c" (UID: "fd84a2fa-265f-442f-ad82-40be74f3c00c"). InnerVolumeSpecName "kube-api-access-qqdcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:57:01.601069 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.601045    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-token" pod "fd84a2fa-265f-442f-ad82-40be74f3c00c" (UID: "fd84a2fa-265f-442f-ad82-40be74f3c00c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:57:01.601457 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.601424    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-podinfo" pod "fd84a2fa-265f-442f-ad82-40be74f3c00c" (UID: "fd84a2fa-265f-442f-ad82-40be74f3c00c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:57:01.700114 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.700081    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqdcj\" (UniqueName: \"kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-kube-api-access-qqdcj\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:01.700114 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.700111    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:01.700114 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.700120    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:01.700362 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:01.700134    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd84a2fa-265f-442f-ad82-40be74f3c00c-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:02.024945 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.024866    4898 generic.go:356] "Generic (PLEG): container finished" podID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerID="3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5" exitCode=0
Nov 02 07:57:02.024945 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.024945    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"
Nov 02 07:57:02.025243 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.024951    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" event={"ID":"fd84a2fa-265f-442f-ad82-40be74f3c00c","Type":"ContainerDied","Data":"3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5"}
Nov 02 07:57:02.025243 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.025010    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4" event={"ID":"fd84a2fa-265f-442f-ad82-40be74f3c00c","Type":"ContainerDied","Data":"6969ce6f92d8306f7476c9355c29e5177861f642b3007a94ccb55e4f0e6ad2ed"}
Nov 02 07:57:02.025243 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.025033    4898 scope.go:122] "RemoveContainer" containerID="3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5"
Nov 02 07:57:02.037184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.037161    4898 scope.go:122] "RemoveContainer" containerID="9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65"
Nov 02 07:57:02.048311 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.048284    4898 scope.go:122] "RemoveContainer" containerID="3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5"
Nov 02 07:57:02.048623 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:57:02.048603    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5\": container with ID starting with 3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5 not found: ID does not exist" containerID="3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5"
Nov 02 07:57:02.048759 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.048632    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5"} err="failed to get container status \"3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5\": rpc error: code = NotFound desc = could not find container \"3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5\": container with ID starting with 3e85aad0430cea35be516cbba727e5db62fec4c45e1870000f4db587d4d2cdd5 not found: ID does not exist"
Nov 02 07:57:02.048759 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.048650    4898 scope.go:122] "RemoveContainer" containerID="9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65"
Nov 02 07:57:02.048995 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:57:02.048964    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65\": container with ID starting with 9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65 not found: ID does not exist" containerID="9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65"
Nov 02 07:57:02.049048 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.049002    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65"} err="failed to get container status \"9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65\": rpc error: code = NotFound desc = could not find container \"9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65\": container with ID starting with 9c451fe7b05f79ff2ef367046f895dbe2b07fe0599f50e1ad40813d0c7f25c65 not found: ID does not exist"
Nov 02 07:57:02.069176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.069149    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"]
Nov 02 07:57:02.099490 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.099451    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"]
Nov 02 07:57:02.104710 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.104684    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"]
Nov 02 07:57:02.111436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:02.111411    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7a27f-openshift-default-549db5f865-2d6f4"]
Nov 02 07:57:03.469378 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:03.469338    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" path="/var/lib/kubelet/pods/fd84a2fa-265f-442f-ad82-40be74f3c00c/volumes"
Nov 02 07:57:04.033671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:04.033636    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"]
Nov 02 07:57:04.045089 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:04.045052    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0182346-9zjv6"]
Nov 02 07:57:05.468328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:05.468283    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e003706c-d239-45f3-8918-a6db4b2505f9" path="/var/lib/kubelet/pods/e003706c-d239-45f3-8918-a6db4b2505f9/volumes"
Nov 02 07:57:06.032806 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:06.032769    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"]
Nov 02 07:57:06.044634 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:06.044607    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d28f2e0-r9gwk"]
Nov 02 07:57:07.069822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.069753    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f" podUID="d7c82e96-5e12-43ff-aaeb-832e929263e4" containerName="payload-processing" containerID="cri-o://92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee" gracePeriod=25
Nov 02 07:57:07.105995 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.105956    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm" podUID="e28bd504-3ad1-4203-83d9-4d5156ac6390" containerName="payload-pre-processing" containerID="cri-o://5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed" gracePeriod=25
Nov 02 07:57:07.418516 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.418485    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"
Nov 02 07:57:07.462002 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.459982    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"
Nov 02 07:57:07.492135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.492084    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a2055a-e183-410d-9c16-2cc678f20f8e" path="/var/lib/kubelet/pods/66a2055a-e183-410d-9c16-2cc678f20f8e/volumes"
Nov 02 07:57:07.516593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.510948    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c82e96-5e12-43ff-aaeb-832e929263e4-kube-api-access-w499s\" (UniqueName: \"kubernetes.io/projected/d7c82e96-5e12-43ff-aaeb-832e929263e4-kube-api-access-w499s\") pod \"d7c82e96-5e12-43ff-aaeb-832e929263e4\" (UID: \"d7c82e96-5e12-43ff-aaeb-832e929263e4\") "
Nov 02 07:57:07.516593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.511048    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c82e96-5e12-43ff-aaeb-832e929263e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c82e96-5e12-43ff-aaeb-832e929263e4-plugins-config-volume\") pod \"d7c82e96-5e12-43ff-aaeb-832e929263e4\" (UID: \"d7c82e96-5e12-43ff-aaeb-832e929263e4\") "
Nov 02 07:57:07.516593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.511213    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e28bd504-3ad1-4203-83d9-4d5156ac6390-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28bd504-3ad1-4203-83d9-4d5156ac6390-plugins-config-volume\") pod \"e28bd504-3ad1-4203-83d9-4d5156ac6390\" (UID: \"e28bd504-3ad1-4203-83d9-4d5156ac6390\") "
Nov 02 07:57:07.516593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.511243    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28bd504-3ad1-4203-83d9-4d5156ac6390-kube-api-access-ws8bw\" (UniqueName: \"kubernetes.io/projected/e28bd504-3ad1-4203-83d9-4d5156ac6390-kube-api-access-ws8bw\") pod \"e28bd504-3ad1-4203-83d9-4d5156ac6390\" (UID: \"e28bd504-3ad1-4203-83d9-4d5156ac6390\") "
Nov 02 07:57:07.516593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.515002    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c82e96-5e12-43ff-aaeb-832e929263e4-plugins-config-volume" pod "d7c82e96-5e12-43ff-aaeb-832e929263e4" (UID: "d7c82e96-5e12-43ff-aaeb-832e929263e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:57:07.516593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.515377    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28bd504-3ad1-4203-83d9-4d5156ac6390-plugins-config-volume" pod "e28bd504-3ad1-4203-83d9-4d5156ac6390" (UID: "e28bd504-3ad1-4203-83d9-4d5156ac6390"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:57:07.517086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.516654    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c82e96-5e12-43ff-aaeb-832e929263e4-kube-api-access-w499s" pod "d7c82e96-5e12-43ff-aaeb-832e929263e4" (UID: "d7c82e96-5e12-43ff-aaeb-832e929263e4"). InnerVolumeSpecName "kube-api-access-w499s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:57:07.517828 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.517797    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28bd504-3ad1-4203-83d9-4d5156ac6390-kube-api-access-ws8bw" pod "e28bd504-3ad1-4203-83d9-4d5156ac6390" (UID: "e28bd504-3ad1-4203-83d9-4d5156ac6390"). InnerVolumeSpecName "kube-api-access-ws8bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:57:07.613459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.613377    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28bd504-3ad1-4203-83d9-4d5156ac6390-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:07.613459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.613407    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8bw\" (UniqueName: \"kubernetes.io/projected/e28bd504-3ad1-4203-83d9-4d5156ac6390-kube-api-access-ws8bw\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:07.613459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.613433    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w499s\" (UniqueName: \"kubernetes.io/projected/d7c82e96-5e12-43ff-aaeb-832e929263e4-kube-api-access-w499s\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:07.613459 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:07.613442    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c82e96-5e12-43ff-aaeb-832e929263e4-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:08.082720 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.082680    4898 generic.go:356] "Generic (PLEG): container finished" podID="d7c82e96-5e12-43ff-aaeb-832e929263e4" containerID="92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee" exitCode=0
Nov 02 07:57:08.083248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.082737    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"
Nov 02 07:57:08.083248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.082765    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f" event={"ID":"d7c82e96-5e12-43ff-aaeb-832e929263e4","Type":"ContainerDied","Data":"92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee"}
Nov 02 07:57:08.083248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.082818    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f" event={"ID":"d7c82e96-5e12-43ff-aaeb-832e929263e4","Type":"ContainerDied","Data":"4a8cd352c6e6cc43c5bd0863ee1244b191d82e0ccbbf2b6ddef0ab766e103943"}
Nov 02 07:57:08.083248 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.082851    4898 scope.go:122] "RemoveContainer" containerID="92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee"
Nov 02 07:57:08.084634 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.084609    4898 generic.go:356] "Generic (PLEG): container finished" podID="e28bd504-3ad1-4203-83d9-4d5156ac6390" containerID="5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed" exitCode=0
Nov 02 07:57:08.084929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.084878    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm" event={"ID":"e28bd504-3ad1-4203-83d9-4d5156ac6390","Type":"ContainerDied","Data":"5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed"}
Nov 02 07:57:08.085011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.084958    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm" event={"ID":"e28bd504-3ad1-4203-83d9-4d5156ac6390","Type":"ContainerDied","Data":"4de6e6ef73aaf465925823405599a0d355b3102c687a877f4377d62b4c8eb58c"}
Nov 02 07:57:08.085071 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.085043    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"
Nov 02 07:57:08.100239 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.100219    4898 scope.go:122] "RemoveContainer" containerID="92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee"
Nov 02 07:57:08.100545 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:57:08.100526    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee\": container with ID starting with 92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee not found: ID does not exist" containerID="92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee"
Nov 02 07:57:08.100610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.100554    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee"} err="failed to get container status \"92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee\": rpc error: code = NotFound desc = could not find container \"92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee\": container with ID starting with 92ce26bf5bf12ae1693579480fb30a6e37d51e9ed10c4de478a150ab9ad939ee not found: ID does not exist"
Nov 02 07:57:08.100610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.100572    4898 scope.go:122] "RemoveContainer" containerID="5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed"
Nov 02 07:57:08.111351 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.111335    4898 scope.go:122] "RemoveContainer" containerID="5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed"
Nov 02 07:57:08.111604 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:57:08.111577    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed\": container with ID starting with 5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed not found: ID does not exist" containerID="5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed"
Nov 02 07:57:08.111668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.111616    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed"} err="failed to get container status \"5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed\": rpc error: code = NotFound desc = could not find container \"5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed\": container with ID starting with 5b3e3a50e21b6f68d59d5852b411dc2e56d59066465c5e0dd9bd0fefadd1ffed not found: ID does not exist"
Nov 02 07:57:08.138828 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.138801    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"]
Nov 02 07:57:08.150393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.150371    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-680c01-df649b747-mdxbm"]
Nov 02 07:57:08.178058 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.178034    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"]
Nov 02 07:57:08.186768 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:08.186743    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-680c01-784857fb69-jhn2f"]
Nov 02 07:57:09.524721 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:09.524682    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c82e96-5e12-43ff-aaeb-832e929263e4" path="/var/lib/kubelet/pods/d7c82e96-5e12-43ff-aaeb-832e929263e4/volumes"
Nov 02 07:57:09.542051 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:09.542017    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e28bd504-3ad1-4203-83d9-4d5156ac6390" path="/var/lib/kubelet/pods/e28bd504-3ad1-4203-83d9-4d5156ac6390/volumes"
Nov 02 07:57:10.896907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:10.896841    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"]
Nov 02 07:57:10.897389 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:10.897137    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy" containerID="cri-o://b8618d5f939b66fa9a2a091bb922ca499a1df4c8b141baa0d1eefb934fa01f3f" gracePeriod=30
Nov 02 07:57:11.543261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:11.543230    4898 scope.go:122] "RemoveContainer" containerID="71c3936fa79d7fa77d701ac20e609691ab33fee3ec2c22c8317a082a0b3a55c0"
Nov 02 07:57:11.554036 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:11.554011    4898 scope.go:122] "RemoveContainer" containerID="f7bd7aa017d08e4d69988e57983bbfb568b20f7cd6461b0408f621364da5d3fc"
Nov 02 07:57:11.566648 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:11.566621    4898 scope.go:122] "RemoveContainer" containerID="d4f9001281526a2977276904a74e28de12998efcfb6f5f2332d729d2a3b01235"
Nov 02 07:57:11.578327 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:11.578300    4898 scope.go:122] "RemoveContainer" containerID="d468986da821c245b657ea0b67ee7993663e616add4691625c7d3ab8b4387545"
Nov 02 07:57:11.588440 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:11.588415    4898 scope.go:122] "RemoveContainer" containerID="c83fb64c0a9f89c2f5d32dc232b9cc9eb5957b07909f36f584ee4f243fc7874c"
Nov 02 07:57:11.598577 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:11.598555    4898 scope.go:122] "RemoveContainer" containerID="75a2c373aabac59de356286ef11de0eda864443936490f6b98c8a0503bdc087a"
Nov 02 07:57:11.609564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:11.609542    4898 scope.go:122] "RemoveContainer" containerID="4a5c7db07060b64a2843a8cdcb1ac3aa06eafc3f035e23957468332647d0a34c"
Nov 02 07:57:11.621861 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:11.621835    4898 scope.go:122] "RemoveContainer" containerID="871c378caba3e82e58608a86840b4fcea561236f21195a3101a6685d41fbf166"
Nov 02 07:57:12.261987 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.261950    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"]
Nov 02 07:57:12.262977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.262954    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerName="istio-proxy"
Nov 02 07:57:12.262977 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.262976    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerName="istio-proxy"
Nov 02 07:57:12.263237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.262994    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d7c82e96-5e12-43ff-aaeb-832e929263e4" containerName="payload-processing"
Nov 02 07:57:12.263237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.263003    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c82e96-5e12-43ff-aaeb-832e929263e4" containerName="payload-processing"
Nov 02 07:57:12.263237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.263025    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e28bd504-3ad1-4203-83d9-4d5156ac6390" containerName="payload-pre-processing"
Nov 02 07:57:12.263237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.263035    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28bd504-3ad1-4203-83d9-4d5156ac6390" containerName="payload-pre-processing"
Nov 02 07:57:12.263237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.263090    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c899bc58-259d-42e1-a77e-8b3c26c7993e" containerName="payload-pre-processing"
Nov 02 07:57:12.263237 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.263098    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c899bc58-259d-42e1-a77e-8b3c26c7993e" containerName="payload-pre-processing"
Nov 02 07:57:12.263593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.263263    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c899bc58-259d-42e1-a77e-8b3c26c7993e" containerName="payload-pre-processing"
Nov 02 07:57:12.263593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.263279    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerName="istio-proxy"
Nov 02 07:57:12.263593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.263289    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerName="istio-proxy"
Nov 02 07:57:12.263593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.263297    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c82e96-5e12-43ff-aaeb-832e929263e4" containerName="payload-processing"
Nov 02 07:57:12.263593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.263307    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28bd504-3ad1-4203-83d9-4d5156ac6390" containerName="payload-pre-processing"
Nov 02 07:57:12.270502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.270477    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.298100 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.298065    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"]
Nov 02 07:57:12.353168 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.353138    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-ac716b-openshift-default-dockercfg-4vwrl\""
Nov 02 07:57:12.423942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.419660    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d8dd92-580e-4e7c-9114-1215f2100400-istiod-ca-cert\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.423942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.419741    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-data\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.423942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.419965    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn7h6\" (UniqueName: \"kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-kube-api-access-sn7h6\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.423942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.420074    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d8dd92-580e-4e7c-9114-1215f2100400-istio-podinfo\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.423942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.420134    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-envoy\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.423942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.420208    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-certs\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.423942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.420243    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-istio-token\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.423942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.420330    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-credential-socket\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.423942 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.420366    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-socket\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.557850 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.543851    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn7h6\" (UniqueName: \"kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-kube-api-access-sn7h6\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.558261 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.558207    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d8dd92-580e-4e7c-9114-1215f2100400-istio-podinfo\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.558418 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.558300    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-envoy\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.558418 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.558378    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-certs\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.558418 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.558412    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-istio-token\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.558595 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.558489    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-credential-socket\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.558595 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.558528    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-socket\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.558717 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.558603    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d8dd92-580e-4e7c-9114-1215f2100400-istiod-ca-cert\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.558717 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.558657    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-data\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.559727 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.559221    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-data\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.576911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.570309    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-socket\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.576911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.570583    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-credential-socket\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.576911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.571226    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d8dd92-580e-4e7c-9114-1215f2100400-istiod-ca-cert\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.576911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.572330    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-certs\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.617912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.616055    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn7h6\" (UniqueName: \"kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-kube-api-access-sn7h6\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.619099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.619072    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-envoy\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.635929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.634909    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-istio-token\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.635929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.635730    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d8dd92-580e-4e7c-9114-1215f2100400-istio-podinfo\") pod \"e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") " pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:12.899804 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:12.899699    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:13.217167 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:57:13.217123    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d8dd92_580e_4e7c_9114_1215f2100400.slice/crio-c05b7315e4905b3387e42e40cab6d896713f792f91b5d0701f4e0c7f1fb7663c WatchSource:0}: Error finding container c05b7315e4905b3387e42e40cab6d896713f792f91b5d0701f4e0c7f1fb7663c: Status 404 returned error can't find the container with id c05b7315e4905b3387e42e40cab6d896713f792f91b5d0701f4e0c7f1fb7663c
Nov 02 07:57:13.218150 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:13.218122    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"]
Nov 02 07:57:13.219812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:13.219778    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:57:13.219899 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:13.219844    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:57:13.219946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:13.219909    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:57:14.176827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:14.176752    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" event={"ID":"54d8dd92-580e-4e7c-9114-1215f2100400","Type":"ContainerStarted","Data":"a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327"}
Nov 02 07:57:14.177255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:14.176839    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" event={"ID":"54d8dd92-580e-4e7c-9114-1215f2100400","Type":"ContainerStarted","Data":"c05b7315e4905b3387e42e40cab6d896713f792f91b5d0701f4e0c7f1fb7663c"}
Nov 02 07:57:14.236178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:14.236119    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" podStartSLOduration=2.2361018440000002 podStartE2EDuration="2.236101844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:57:14.234939777 +0000 UTC m=+1984.925266763" watchObservedRunningTime="2026-11-02 07:57:14.236101844 +0000 UTC m=+1984.926428808"
Nov 02 07:57:14.900762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:14.900720    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:14.906560 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:14.906527    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:14.993077 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:57:14.993039    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:57:15.184664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:15.184578    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:15.185630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:15.185609    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:57:16.035808 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.035775    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"]
Nov 02 07:57:16.044926 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.044884    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeffbe7f-g86hn"]
Nov 02 07:57:16.197801 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.197759    4898 generic.go:356] "Generic (PLEG): container finished" podID="bef52159-5187-4636-8278-b7405428e66d" containerID="b8618d5f939b66fa9a2a091bb922ca499a1df4c8b141baa0d1eefb934fa01f3f" exitCode=0
Nov 02 07:57:16.198208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.197834    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" event={"ID":"bef52159-5187-4636-8278-b7405428e66d","Type":"ContainerDied","Data":"b8618d5f939b66fa9a2a091bb922ca499a1df4c8b141baa0d1eefb934fa01f3f"}
Nov 02 07:57:16.198208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.197932    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt" event={"ID":"bef52159-5187-4636-8278-b7405428e66d","Type":"ContainerDied","Data":"79ef5444e5adbbb3ba6b1b33ff2363ad182774c9c083979d0044829de93537bf"}
Nov 02 07:57:16.198208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.197961    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79ef5444e5adbbb3ba6b1b33ff2363ad182774c9c083979d0044829de93537bf"
Nov 02 07:57:16.198208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.197992    4898 scope.go:122] "RemoveContainer" containerID="6b4c6504f7a6dfbfc567b5e5cd3b7657013291a20e9c02f6fb6375c754983200"
Nov 02 07:57:16.201044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.201023    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:57:16.231371 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231345    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-socket\") pod \"bef52159-5187-4636-8278-b7405428e66d\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") "
Nov 02 07:57:16.231477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231379    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bef52159-5187-4636-8278-b7405428e66d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef52159-5187-4636-8278-b7405428e66d-istio-podinfo\") pod \"bef52159-5187-4636-8278-b7405428e66d\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") "
Nov 02 07:57:16.231477 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231461    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-credential-socket\") pod \"bef52159-5187-4636-8278-b7405428e66d\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") "
Nov 02 07:57:16.231585 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231504    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-certs\") pod \"bef52159-5187-4636-8278-b7405428e66d\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") "
Nov 02 07:57:16.231585 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231559    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-data\") pod \"bef52159-5187-4636-8278-b7405428e66d\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") "
Nov 02 07:57:16.231680 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231624    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-envoy\") pod \"bef52159-5187-4636-8278-b7405428e66d\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") "
Nov 02 07:57:16.231680 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231624    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-socket" pod "bef52159-5187-4636-8278-b7405428e66d" (UID: "bef52159-5187-4636-8278-b7405428e66d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:57:16.231680 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231648    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-istio-token\" (UniqueName: \"kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-istio-token\") pod \"bef52159-5187-4636-8278-b7405428e66d\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") "
Nov 02 07:57:16.231835 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231720    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-kube-api-access-d9wz8\" (UniqueName: \"kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-kube-api-access-d9wz8\") pod \"bef52159-5187-4636-8278-b7405428e66d\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") "
Nov 02 07:57:16.231938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231838    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef52159-5187-4636-8278-b7405428e66d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef52159-5187-4636-8278-b7405428e66d-istiod-ca-cert\") pod \"bef52159-5187-4636-8278-b7405428e66d\" (UID: \"bef52159-5187-4636-8278-b7405428e66d\") "
Nov 02 07:57:16.231938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231837    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-certs" pod "bef52159-5187-4636-8278-b7405428e66d" (UID: "bef52159-5187-4636-8278-b7405428e66d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:57:16.232110 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231864    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-credential-socket" pod "bef52159-5187-4636-8278-b7405428e66d" (UID: "bef52159-5187-4636-8278-b7405428e66d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:57:16.232110 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.231883    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-data" pod "bef52159-5187-4636-8278-b7405428e66d" (UID: "bef52159-5187-4636-8278-b7405428e66d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:57:16.232350 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.232327    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef52159-5187-4636-8278-b7405428e66d-istiod-ca-cert" pod "bef52159-5187-4636-8278-b7405428e66d" (UID: "bef52159-5187-4636-8278-b7405428e66d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:57:16.232688 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.232665    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:16.232782 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.232694    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef52159-5187-4636-8278-b7405428e66d-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:16.232782 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.232716    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:16.232782 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.232734    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:16.232782 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.232753    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:16.234141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.234112    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bef52159-5187-4636-8278-b7405428e66d-istio-podinfo" pod "bef52159-5187-4636-8278-b7405428e66d" (UID: "bef52159-5187-4636-8278-b7405428e66d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:57:16.234228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.234205    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-istio-token" pod "bef52159-5187-4636-8278-b7405428e66d" (UID: "bef52159-5187-4636-8278-b7405428e66d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:57:16.234467 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.234446    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-kube-api-access-d9wz8" pod "bef52159-5187-4636-8278-b7405428e66d" (UID: "bef52159-5187-4636-8278-b7405428e66d"). InnerVolumeSpecName "kube-api-access-d9wz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:57:16.234689 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.234668    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-envoy" pod "bef52159-5187-4636-8278-b7405428e66d" (UID: "bef52159-5187-4636-8278-b7405428e66d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:57:16.333813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.333772    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef52159-5187-4636-8278-b7405428e66d-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:16.333813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.333804    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef52159-5187-4636-8278-b7405428e66d-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:16.333813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.333817    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:16.334086 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:16.333831    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9wz8\" (UniqueName: \"kubernetes.io/projected/bef52159-5187-4636-8278-b7405428e66d-kube-api-access-d9wz8\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:57:17.206929 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:17.206873    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"
Nov 02 07:57:17.317106 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:17.317073    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"]
Nov 02 07:57:17.320997 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:17.320974    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-680c01-openshift-default-7fcbcfd5-bqwtt"]
Nov 02 07:57:17.466433 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:17.466350    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d20daab-f0d4-4f9c-ac1d-70bca6c37618" path="/var/lib/kubelet/pods/4d20daab-f0d4-4f9c-ac1d-70bca6c37618/volumes"
Nov 02 07:57:17.467017 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:17.466994    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef52159-5187-4636-8278-b7405428e66d" path="/var/lib/kubelet/pods/bef52159-5187-4636-8278-b7405428e66d/volumes"
Nov 02 07:57:18.516691 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.516662    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"]
Nov 02 07:57:18.517393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.517372    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy"
Nov 02 07:57:18.517436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.517396    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy"
Nov 02 07:57:18.517436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.517416    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy"
Nov 02 07:57:18.517436 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.517421    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy"
Nov 02 07:57:18.517536 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.517437    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerName="istio-proxy"
Nov 02 07:57:18.517536 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.517442    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd84a2fa-265f-442f-ad82-40be74f3c00c" containerName="istio-proxy"
Nov 02 07:57:18.517594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.517556    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy"
Nov 02 07:57:18.523343 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.523322    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"
Nov 02 07:57:18.543090 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.543064    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"]
Nov 02 07:57:18.549698 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.549671    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-ac716b\""
Nov 02 07:57:18.549818 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.549728    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-ac716b-dockercfg-lc9xb\""
Nov 02 07:57:18.571707 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.571671    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969f6f63-470f-4b22-b80b-cc3f33fb6f14-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf\" (UID: \"969f6f63-470f-4b22-b80b-cc3f33fb6f14\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"
Nov 02 07:57:18.571912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.571767    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl8rx\" (UniqueName: \"kubernetes.io/projected/969f6f63-470f-4b22-b80b-cc3f33fb6f14-kube-api-access-tl8rx\") pod \"payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf\" (UID: \"969f6f63-470f-4b22-b80b-cc3f33fb6f14\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"
Nov 02 07:57:18.706067 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.705202    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"]
Nov 02 07:57:18.706597 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.706570    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef52159-5187-4636-8278-b7405428e66d" containerName="istio-proxy"
Nov 02 07:57:18.817514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.817409    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969f6f63-470f-4b22-b80b-cc3f33fb6f14-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf\" (UID: \"969f6f63-470f-4b22-b80b-cc3f33fb6f14\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"
Nov 02 07:57:18.817514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.817502    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl8rx\" (UniqueName: \"kubernetes.io/projected/969f6f63-470f-4b22-b80b-cc3f33fb6f14-kube-api-access-tl8rx\") pod \"payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf\" (UID: \"969f6f63-470f-4b22-b80b-cc3f33fb6f14\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"
Nov 02 07:57:18.821055 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.821004    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969f6f63-470f-4b22-b80b-cc3f33fb6f14-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf\" (UID: \"969f6f63-470f-4b22-b80b-cc3f33fb6f14\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"
Nov 02 07:57:18.824542 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.824518    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"]
Nov 02 07:57:18.824812 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.824752    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"
Nov 02 07:57:18.863880 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.845316    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl8rx\" (UniqueName: \"kubernetes.io/projected/969f6f63-470f-4b22-b80b-cc3f33fb6f14-kube-api-access-tl8rx\") pod \"payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf\" (UID: \"969f6f63-470f-4b22-b80b-cc3f33fb6f14\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"
Nov 02 07:57:18.919556 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.919523    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l\" (UID: \"ff6ee21d-7956-4d2c-b00a-8e38afe9c585\") " pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"
Nov 02 07:57:18.919739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:18.919723    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g967m\" (UniqueName: \"kubernetes.io/projected/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-kube-api-access-g967m\") pod \"payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l\" (UID: \"ff6ee21d-7956-4d2c-b00a-8e38afe9c585\") " pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"
Nov 02 07:57:19.020681 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:19.020647    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l\" (UID: \"ff6ee21d-7956-4d2c-b00a-8e38afe9c585\") " pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"
Nov 02 07:57:19.020867 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:19.020789    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g967m\" (UniqueName: \"kubernetes.io/projected/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-kube-api-access-g967m\") pod \"payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l\" (UID: \"ff6ee21d-7956-4d2c-b00a-8e38afe9c585\") " pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"
Nov 02 07:57:19.021561 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:19.021535    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l\" (UID: \"ff6ee21d-7956-4d2c-b00a-8e38afe9c585\") " pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"
Nov 02 07:57:19.044737 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:19.044706    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g967m\" (UniqueName: \"kubernetes.io/projected/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-kube-api-access-g967m\") pod \"payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l\" (UID: \"ff6ee21d-7956-4d2c-b00a-8e38afe9c585\") " pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"
Nov 02 07:57:19.137593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:19.137497    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"
Nov 02 07:57:19.180158 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:19.180117    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"
Nov 02 07:57:19.448872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:19.448314    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"]
Nov 02 07:57:19.449124 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:57:19.449087    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969f6f63_470f_4b22_b80b_cc3f33fb6f14.slice/crio-8655ffd7b9f6253cdfcd26636fb978b7a65aa6d3b184df5b11d7d05dcba3bf64 WatchSource:0}: Error finding container 8655ffd7b9f6253cdfcd26636fb978b7a65aa6d3b184df5b11d7d05dcba3bf64: Status 404 returned error can't find the container with id 8655ffd7b9f6253cdfcd26636fb978b7a65aa6d3b184df5b11d7d05dcba3bf64
Nov 02 07:57:19.520125 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:57:19.520093    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6ee21d_7956_4d2c_b00a_8e38afe9c585.slice/crio-4b48cfae6d04d75dc57ff04abfbb148801e776c7610c2f2b54497ab0d0b0cdb4 WatchSource:0}: Error finding container 4b48cfae6d04d75dc57ff04abfbb148801e776c7610c2f2b54497ab0d0b0cdb4: Status 404 returned error can't find the container with id 4b48cfae6d04d75dc57ff04abfbb148801e776c7610c2f2b54497ab0d0b0cdb4
Nov 02 07:57:19.523366 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:19.523344    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"]
Nov 02 07:57:20.032145 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:20.032103    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"]
Nov 02 07:57:20.036144 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:20.036107    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5145011a-tlsbs"]
Nov 02 07:57:20.237704 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:20.237663    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf" event={"ID":"969f6f63-470f-4b22-b80b-cc3f33fb6f14","Type":"ContainerStarted","Data":"0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7"}
Nov 02 07:57:20.238035 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:20.238007    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf" event={"ID":"969f6f63-470f-4b22-b80b-cc3f33fb6f14","Type":"ContainerStarted","Data":"8655ffd7b9f6253cdfcd26636fb978b7a65aa6d3b184df5b11d7d05dcba3bf64"}
Nov 02 07:57:20.238172 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:20.238045    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"
Nov 02 07:57:20.239661 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:20.239626    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l" event={"ID":"ff6ee21d-7956-4d2c-b00a-8e38afe9c585","Type":"ContainerStarted","Data":"963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859"}
Nov 02 07:57:20.239788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:20.239686    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l" event={"ID":"ff6ee21d-7956-4d2c-b00a-8e38afe9c585","Type":"ContainerStarted","Data":"4b48cfae6d04d75dc57ff04abfbb148801e776c7610c2f2b54497ab0d0b0cdb4"}
Nov 02 07:57:20.239788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:20.239711    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"
Nov 02 07:57:20.279822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:20.279771    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf" podStartSLOduration=2.279757975 podStartE2EDuration="2.279757975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:57:20.278238253 +0000 UTC m=+1990.968565218" watchObservedRunningTime="2026-11-02 07:57:20.279757975 +0000 UTC m=+1990.970084938"
Nov 02 07:57:20.321552 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:20.321475    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l" podStartSLOduration=2.321462182 podStartE2EDuration="2.321462182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:57:20.318340801 +0000 UTC m=+1991.008667784" watchObservedRunningTime="2026-11-02 07:57:20.321462182 +0000 UTC m=+1991.011789146"
Nov 02 07:57:21.468503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:21.468462    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7df9796-fb9b-451d-b366-559ecf56a6ab" path="/var/lib/kubelet/pods/b7df9796-fb9b-451d-b366-559ecf56a6ab/volumes"
Nov 02 07:57:25.031644 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:25.031609    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"]
Nov 02 07:57:25.043701 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:25.043666    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed387da-4qkch"]
Nov 02 07:57:25.466648 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:25.466606    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c141c40-6b3e-4c57-9810-0b0f1987c2d8" path="/var/lib/kubelet/pods/3c141c40-6b3e-4c57-9810-0b0f1987c2d8/volumes"
Nov 02 07:57:29.035620 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:29.035582    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"]
Nov 02 07:57:29.044776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:29.044742    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a9e5c3-9lz4s"]
Nov 02 07:57:29.468164 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:29.468123    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38878f90-ab7b-4eb3-ad7b-0f600288e498" path="/var/lib/kubelet/pods/38878f90-ab7b-4eb3-ad7b-0f600288e498/volumes"
Nov 02 07:57:31.252159 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:31.252112    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"
Nov 02 07:57:31.252589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:31.252197    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"
Nov 02 07:57:40.034730 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:40.034693    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"]
Nov 02 07:57:40.046790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:40.046761    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc5365d8-hrgc6"]
Nov 02 07:57:41.036748 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:41.036700    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"]
Nov 02 07:57:41.049300 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:41.049264    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a36ede-55fhv"]
Nov 02 07:57:41.466014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:41.465979    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81edb37-6116-4ca9-aa4f-06283464d900" path="/var/lib/kubelet/pods/c81edb37-6116-4ca9-aa4f-06283464d900/volumes"
Nov 02 07:57:41.466515 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:57:41.466500    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce951366-9ab8-4843-8b17-e9ddb906df18" path="/var/lib/kubelet/pods/ce951366-9ab8-4843-8b17-e9ddb906df18/volumes"
Nov 02 07:57:44.991075 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:57:44.991035    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:58:02.481187 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:02.481104    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"]
Nov 02 07:58:02.481706 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:02.481374    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s" podUID="687d6399-45b2-48ea-9290-bf675a2b22bb" containerName="istio-proxy" containerID="cri-o://a19d2637adbe3ed64a57472a9d9def34ab0878627922a2efae004a4fd996cdeb" gracePeriod=30
Nov 02 07:58:02.494964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:02.494932    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"]
Nov 02 07:58:02.495220 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:02.495176    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss" podUID="fd3bbf16-61be-4148-b7ff-5c862aed49d1" containerName="istio-proxy" containerID="cri-o://d424c7f29b35eacb4cc35208b15b06a35fbe91d494b30c73d84bae87a16b7fe4" gracePeriod=30
Nov 02 07:58:07.878914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:07.856629    4898 generic.go:356] "Generic (PLEG): container finished" podID="fd3bbf16-61be-4148-b7ff-5c862aed49d1" containerID="d424c7f29b35eacb4cc35208b15b06a35fbe91d494b30c73d84bae87a16b7fe4" exitCode=0
Nov 02 07:58:07.878914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:07.856803    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss" event={"ID":"fd3bbf16-61be-4148-b7ff-5c862aed49d1","Type":"ContainerDied","Data":"d424c7f29b35eacb4cc35208b15b06a35fbe91d494b30c73d84bae87a16b7fe4"}
Nov 02 07:58:07.930213 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:07.930167    4898 generic.go:356] "Generic (PLEG): container finished" podID="687d6399-45b2-48ea-9290-bf675a2b22bb" containerID="a19d2637adbe3ed64a57472a9d9def34ab0878627922a2efae004a4fd996cdeb" exitCode=0
Nov 02 07:58:07.930524 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:07.930487    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s" event={"ID":"687d6399-45b2-48ea-9290-bf675a2b22bb","Type":"ContainerDied","Data":"a19d2637adbe3ed64a57472a9d9def34ab0878627922a2efae004a4fd996cdeb"}
Nov 02 07:58:07.960912 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:07.955030    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:58:08.022536 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.022501    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/687d6399-45b2-48ea-9290-bf675a2b22bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687d6399-45b2-48ea-9290-bf675a2b22bb-istiod-ca-cert\") pod \"687d6399-45b2-48ea-9290-bf675a2b22bb\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") "
Nov 02 07:58:08.022694 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.022638    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-kube-api-access-d5ntb\" (UniqueName: \"kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-kube-api-access-d5ntb\") pod \"687d6399-45b2-48ea-9290-bf675a2b22bb\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") "
Nov 02 07:58:08.022694 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.022662    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-data\") pod \"687d6399-45b2-48ea-9290-bf675a2b22bb\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") "
Nov 02 07:58:08.022776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.022731    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-certs\") pod \"687d6399-45b2-48ea-9290-bf675a2b22bb\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") "
Nov 02 07:58:08.022825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.022788    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-envoy\") pod \"687d6399-45b2-48ea-9290-bf675a2b22bb\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") "
Nov 02 07:58:08.022874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.022830    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687d6399-45b2-48ea-9290-bf675a2b22bb-istiod-ca-cert" pod "687d6399-45b2-48ea-9290-bf675a2b22bb" (UID: "687d6399-45b2-48ea-9290-bf675a2b22bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:58:08.022874 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.022866    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-credential-socket\") pod \"687d6399-45b2-48ea-9290-bf675a2b22bb\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") "
Nov 02 07:58:08.023024 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.022962    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-socket\") pod \"687d6399-45b2-48ea-9290-bf675a2b22bb\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") "
Nov 02 07:58:08.023084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.023047    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-token\" (UniqueName: \"kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-token\") pod \"687d6399-45b2-48ea-9290-bf675a2b22bb\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") "
Nov 02 07:58:08.023147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.023080    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-data" pod "687d6399-45b2-48ea-9290-bf675a2b22bb" (UID: "687d6399-45b2-48ea-9290-bf675a2b22bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:08.023147 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.023110    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-podinfo\") pod \"687d6399-45b2-48ea-9290-bf675a2b22bb\" (UID: \"687d6399-45b2-48ea-9290-bf675a2b22bb\") "
Nov 02 07:58:08.023254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.023213    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-socket" pod "687d6399-45b2-48ea-9290-bf675a2b22bb" (UID: "687d6399-45b2-48ea-9290-bf675a2b22bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:08.023827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.023800    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.023982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.023830    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687d6399-45b2-48ea-9290-bf675a2b22bb-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.023982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.023833    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-credential-socket" pod "687d6399-45b2-48ea-9290-bf675a2b22bb" (UID: "687d6399-45b2-48ea-9290-bf675a2b22bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:08.023982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.023844    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.025042 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.025000    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-certs" pod "687d6399-45b2-48ea-9290-bf675a2b22bb" (UID: "687d6399-45b2-48ea-9290-bf675a2b22bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:08.025198 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.025019    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:58:08.026522 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.026492    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-podinfo" pod "687d6399-45b2-48ea-9290-bf675a2b22bb" (UID: "687d6399-45b2-48ea-9290-bf675a2b22bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:58:08.026669 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.026546    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-envoy" pod "687d6399-45b2-48ea-9290-bf675a2b22bb" (UID: "687d6399-45b2-48ea-9290-bf675a2b22bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:08.026720 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.026687    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-kube-api-access-d5ntb" pod "687d6399-45b2-48ea-9290-bf675a2b22bb" (UID: "687d6399-45b2-48ea-9290-bf675a2b22bb"). InnerVolumeSpecName "kube-api-access-d5ntb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:08.026836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.026822    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-token" pod "687d6399-45b2-48ea-9290-bf675a2b22bb" (UID: "687d6399-45b2-48ea-9290-bf675a2b22bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:08.125201 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.125109    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-data\") pod \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") "
Nov 02 07:58:08.125382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.125221    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-envoy\") pod \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") "
Nov 02 07:58:08.125382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.125315    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-socket\") pod \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") "
Nov 02 07:58:08.125518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.125397    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-data" pod "fd3bbf16-61be-4148-b7ff-5c862aed49d1" (UID: "fd3bbf16-61be-4148-b7ff-5c862aed49d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:08.125518 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.125411    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-podinfo\") pod \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") "
Nov 02 07:58:08.125632 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.125551    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-kube-api-access-mn9vz\" (UniqueName: \"kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-kube-api-access-mn9vz\") pod \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") "
Nov 02 07:58:08.125632 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.125574    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-socket" pod "fd3bbf16-61be-4148-b7ff-5c862aed49d1" (UID: "fd3bbf16-61be-4148-b7ff-5c862aed49d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:08.125632 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.125611    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istiod-ca-cert\") pod \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") "
Nov 02 07:58:08.125794 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.125696    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-credential-socket\") pod \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") "
Nov 02 07:58:08.125794 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.125755    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-token\" (UniqueName: \"kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-token\") pod \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") "
Nov 02 07:58:08.125915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.125800    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-certs\") pod \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\" (UID: \"fd3bbf16-61be-4148-b7ff-5c862aed49d1\") "
Nov 02 07:58:08.125978 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.125937    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istiod-ca-cert" pod "fd3bbf16-61be-4148-b7ff-5c862aed49d1" (UID: "fd3bbf16-61be-4148-b7ff-5c862aed49d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:58:08.126155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126130    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-credential-socket" pod "fd3bbf16-61be-4148-b7ff-5c862aed49d1" (UID: "fd3bbf16-61be-4148-b7ff-5c862aed49d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:08.126367 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126340    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-certs" pod "fd3bbf16-61be-4148-b7ff-5c862aed49d1" (UID: "fd3bbf16-61be-4148-b7ff-5c862aed49d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:08.126530 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126508    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.126606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126541    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5ntb\" (UniqueName: \"kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-kube-api-access-d5ntb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.126606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126558    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.126606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126573    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.126606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126586    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.126606 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126599    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687d6399-45b2-48ea-9290-bf675a2b22bb-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.126884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126613    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.126884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126626    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.126884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126639    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.126884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126653    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687d6399-45b2-48ea-9290-bf675a2b22bb-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.126884 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.126666    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.128386 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.128357    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-token" pod "fd3bbf16-61be-4148-b7ff-5c862aed49d1" (UID: "fd3bbf16-61be-4148-b7ff-5c862aed49d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:08.128507 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.128492    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-podinfo" pod "fd3bbf16-61be-4148-b7ff-5c862aed49d1" (UID: "fd3bbf16-61be-4148-b7ff-5c862aed49d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:58:08.129231 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.129208    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-kube-api-access-mn9vz" pod "fd3bbf16-61be-4148-b7ff-5c862aed49d1" (UID: "fd3bbf16-61be-4148-b7ff-5c862aed49d1"). InnerVolumeSpecName "kube-api-access-mn9vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:08.129414 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.129394    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-envoy" pod "fd3bbf16-61be-4148-b7ff-5c862aed49d1" (UID: "fd3bbf16-61be-4148-b7ff-5c862aed49d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:08.227575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.227532    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.227575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.227567    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn9vz\" (UniqueName: \"kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-kube-api-access-mn9vz\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.227575 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.227583    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.227825 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.227596    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3bbf16-61be-4148-b7ff-5c862aed49d1-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:08.943960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.943930    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"
Nov 02 07:58:08.944463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.943928    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s" event={"ID":"687d6399-45b2-48ea-9290-bf675a2b22bb","Type":"ContainerDied","Data":"5af7c08aae6ee7a6148dad43e0441161e419a9e57cbd9aa425673bf8b428ecbc"}
Nov 02 07:58:08.944463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.944083    4898 scope.go:122] "RemoveContainer" containerID="a19d2637adbe3ed64a57472a9d9def34ab0878627922a2efae004a4fd996cdeb"
Nov 02 07:58:08.950139 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.950117    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"
Nov 02 07:58:08.950320 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.950115    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss" event={"ID":"fd3bbf16-61be-4148-b7ff-5c862aed49d1","Type":"ContainerDied","Data":"0bec73b8b2b683728bcf8c1718df52b6a3c3ed6850f0700787c14eecb576bb7f"}
Nov 02 07:58:08.967191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:08.967173    4898 scope.go:122] "RemoveContainer" containerID="d424c7f29b35eacb4cc35208b15b06a35fbe91d494b30c73d84bae87a16b7fe4"
Nov 02 07:58:09.013961 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:09.013916    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"]
Nov 02 07:58:09.020418 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:09.020386    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-mxbss"]
Nov 02 07:58:09.179914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:09.176984    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"]
Nov 02 07:58:09.179914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:09.177062    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qm2s"]
Nov 02 07:58:09.472946 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:09.472907    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="687d6399-45b2-48ea-9290-bf675a2b22bb" path="/var/lib/kubelet/pods/687d6399-45b2-48ea-9290-bf675a2b22bb/volumes"
Nov 02 07:58:09.473636 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:09.473614    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3bbf16-61be-4148-b7ff-5c862aed49d1" path="/var/lib/kubelet/pods/fd3bbf16-61be-4148-b7ff-5c862aed49d1/volumes"
Nov 02 07:58:11.723593 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:11.723558    4898 scope.go:122] "RemoveContainer" containerID="266135aaf286b28731679f610ffffa2068afcfe9fbc329cc8b77ba982231c957"
Nov 02 07:58:11.736697 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:11.736669    4898 scope.go:122] "RemoveContainer" containerID="9ccf854d3bf619bdcf6ad8c88a7750c00fcbd5cb9d1ad9d69c448122bc316a9c"
Nov 02 07:58:11.748583 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:11.748553    4898 scope.go:122] "RemoveContainer" containerID="9b9758aa90ec62c8127a308309a9348c06d74d57309141e922e73953bdaca688"
Nov 02 07:58:11.760614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:11.760592    4898 scope.go:122] "RemoveContainer" containerID="e657d5c3e534e22338e1f90d147ad23cb4773ccba237b377d3076b2aa2ed57b4"
Nov 02 07:58:11.771176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:11.771154    4898 scope.go:122] "RemoveContainer" containerID="96db976464426ba4bf02a2fc99c2444aa2b24e78dbe3e62b6b24286b34740e09"
Nov 02 07:58:11.792728 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:11.792703    4898 scope.go:122] "RemoveContainer" containerID="cab198abef887c957d783a28a4a624637237efecd2909596d89c4c650ca9c321"
Nov 02 07:58:13.020913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:13.015735    4898 generic.go:356] "Generic (PLEG): container finished" podID="54d8dd92-580e-4e7c-9114-1215f2100400" containerID="a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327" exitCode=0
Nov 02 07:58:13.020913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:13.015828    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" event={"ID":"54d8dd92-580e-4e7c-9114-1215f2100400","Type":"ContainerDied","Data":"a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327"}
Nov 02 07:58:13.020913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:13.016400    4898 scope.go:122] "RemoveContainer" containerID="a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327"
Nov 02 07:58:13.020913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:13.019881    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:58:13.020913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:13.019994    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:58:13.020913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:13.020044    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:58:14.026128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.026094    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" event={"ID":"54d8dd92-580e-4e7c-9114-1215f2100400","Type":"ContainerStarted","Data":"c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42"}
Nov 02 07:58:14.208827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.201985    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"]
Nov 02 07:58:14.208827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.202968    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fd3bbf16-61be-4148-b7ff-5c862aed49d1" containerName="istio-proxy"
Nov 02 07:58:14.208827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.202990    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3bbf16-61be-4148-b7ff-5c862aed49d1" containerName="istio-proxy"
Nov 02 07:58:14.208827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.203072    4898 cpu_manager.go:414] "containerMap: removing container" podUID="687d6399-45b2-48ea-9290-bf675a2b22bb" containerName="istio-proxy"
Nov 02 07:58:14.208827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.203081    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="687d6399-45b2-48ea-9290-bf675a2b22bb" containerName="istio-proxy"
Nov 02 07:58:14.208827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.203257    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3bbf16-61be-4148-b7ff-5c862aed49d1" containerName="istio-proxy"
Nov 02 07:58:14.208827 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.203286    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="687d6399-45b2-48ea-9290-bf675a2b22bb" containerName="istio-proxy"
Nov 02 07:58:14.212002 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.211978    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"]
Nov 02 07:58:14.212210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.212115    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.228604 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.228529    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ac716b\"/\"openshift-service-ca.crt\""
Nov 02 07:58:14.228975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.228776    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ac716b\"/\"default-dockercfg-8pwl7\""
Nov 02 07:58:14.228975 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.228874    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ac716b\"/\"kube-root-ca.crt\""
Nov 02 07:58:14.229139 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.228982    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ac716b\"/\"explicit-model-ac716b-kserve-self-signed-certs\""
Nov 02 07:58:14.399020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.397905    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbfmg\" (UniqueName: \"kubernetes.io/projected/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-kube-api-access-gbfmg\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.399020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.398008    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tls-certs\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.399020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.398078    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tmp-dir\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.399020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.398110    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-home\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.399020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.398152    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-model-cache\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.399020 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.398195    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-dshm\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.532536 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.531675    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbfmg\" (UniqueName: \"kubernetes.io/projected/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-kube-api-access-gbfmg\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.532536 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.531806    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tls-certs\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.532536 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.532030    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tmp-dir\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.532536 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.532191    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-home\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.532536 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.532235    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-model-cache\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.532536 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.532394    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-dshm\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.536948 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.533671    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tmp-dir\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.577915 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.555223    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-home\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.578425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.578359    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-model-cache\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.604182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.604145    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbfmg\" (UniqueName: \"kubernetes.io/projected/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-kube-api-access-gbfmg\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.604616 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.604515    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-dshm\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.608101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.608072    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tls-certs\") pod \"explicit-model-ac716b-kserve-c665dbfdc-cdd4s\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") " pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.830208 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.830149    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:14.907116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.907081    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:58:14.907461 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.907438    4898 patch_prober.go:29] interesting pod/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 02 07:58:14.907664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:14.907635    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 02 07:58:15.052923 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:58:15.052389    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:58:15.187013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:15.186978    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:58:15.290913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:15.290270    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"]
Nov 02 07:58:15.290913 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:58:15.290443    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0389b0c4_5ad0_4a32_982c_e36ca577f3ac.slice/crio-2abc9e42536523814b478240cd81b15bd184bc060e21464682dd37205b07a508 WatchSource:0}: Error finding container 2abc9e42536523814b478240cd81b15bd184bc060e21464682dd37205b07a508: Status 404 returned error can't find the container with id 2abc9e42536523814b478240cd81b15bd184bc060e21464682dd37205b07a508
Nov 02 07:58:15.900324 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:15.900288    4898 patch_prober.go:29] interesting pod/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 02 07:58:15.900569 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:15.900351    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 02 07:58:16.073060 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:16.073022    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s" event={"ID":"0389b0c4-5ad0-4a32-982c-e36ca577f3ac","Type":"ContainerStarted","Data":"7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694"}
Nov 02 07:58:16.073621 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:16.073591    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s" event={"ID":"0389b0c4-5ad0-4a32-982c-e36ca577f3ac","Type":"ContainerStarted","Data":"2abc9e42536523814b478240cd81b15bd184bc060e21464682dd37205b07a508"}
Nov 02 07:58:16.120226 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:16.120174    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s" podStartSLOduration=2.120161873 podStartE2EDuration="2.120161873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:58:16.11957662 +0000 UTC m=+2046.809903604" watchObservedRunningTime="2026-11-02 07:58:16.120161873 +0000 UTC m=+2046.810488834"
Nov 02 07:58:16.831107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:16.831069    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:16.901079 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:16.900457    4898 patch_prober.go:29] interesting pod/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 02 07:58:16.901079 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:16.900518    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 02 07:58:17.900328 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:17.900287    4898 patch_prober.go:29] interesting pod/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 02 07:58:17.900739 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:17.900349    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 02 07:58:18.900967 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:18.900923    4898 patch_prober.go:29] interesting pod/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 02 07:58:18.901473 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:18.900989    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 02 07:58:19.906011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:19.905972    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:58:19.907241 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:19.907215    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:58:24.831118 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:24.831080    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:24.844822 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:24.844796    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:24.894342 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:24.894306    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"]
Nov 02 07:58:24.894552 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:24.894531    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4" podUID="1409db03-535a-4343-902e-bcfcadd91650" containerName="istio-proxy" containerID="cri-o://819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b" gracePeriod=30
Nov 02 07:58:24.914774 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:24.914741    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"]
Nov 02 07:58:24.915073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:24.915038    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5" podUID="1127b479-5f27-43d2-a86b-069a7cd84d39" containerName="istio-proxy" containerID="cri-o://ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97" gracePeriod=30
Nov 02 07:58:25.181317 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:25.181285    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:27.239228 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.187936    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"]
Nov 02 07:58:27.269074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.267742    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"]
Nov 02 07:58:27.269074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.267849    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:27.283089 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.283066    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:58:27.285040 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.284999    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:58:27.285401 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.285386    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:58:27.296142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.296083    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:58:27.364821 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.364693    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2g8w\" (UniqueName: \"kubernetes.io/projected/1ab85ca9-578d-4911-ab8f-89f8505ddd40-kube-api-access-m2g8w\") pod \"maas-api-revoke-sub-36ff88e3-6x24f\" (UID: \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:27.365099 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.364877    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ff88e3-6x24f\" (UID: \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:27.365196 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.365112    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ff88e3-6x24f\" (UID: \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:27.471910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.468735    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ff88e3-6x24f\" (UID: \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:27.471910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.468956    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2g8w\" (UniqueName: \"kubernetes.io/projected/1ab85ca9-578d-4911-ab8f-89f8505ddd40-kube-api-access-m2g8w\") pod \"maas-api-revoke-sub-36ff88e3-6x24f\" (UID: \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:27.471910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.469074    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ff88e3-6x24f\" (UID: \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:27.471910 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.469983    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ff88e3-6x24f\" (UID: \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:27.484296 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.481553    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ff88e3-6x24f\" (UID: \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:27.497611 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.497530    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2g8w\" (UniqueName: \"kubernetes.io/projected/1ab85ca9-578d-4911-ab8f-89f8505ddd40-kube-api-access-m2g8w\") pod \"maas-api-revoke-sub-36ff88e3-6x24f\" (UID: \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:27.623108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.623069    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:27.941388 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:58:27.941360    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab85ca9_578d_4911_ab8f_89f8505ddd40.slice/crio-e6cfe25fc45c99a9a654ed15623333318761e23e26043a4705aac487ee76ebb7 WatchSource:0}: Error finding container e6cfe25fc45c99a9a654ed15623333318761e23e26043a4705aac487ee76ebb7: Status 404 returned error can't find the container with id e6cfe25fc45c99a9a654ed15623333318761e23e26043a4705aac487ee76ebb7
Nov 02 07:58:27.943183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:27.943110    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"]
Nov 02 07:58:28.278503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:28.278466    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f" event={"ID":"1ab85ca9-578d-4911-ab8f-89f8505ddd40","Type":"ContainerStarted","Data":"e6cfe25fc45c99a9a654ed15623333318761e23e26043a4705aac487ee76ebb7"}
Nov 02 07:58:29.033490 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:29.033447    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"]
Nov 02 07:58:29.045502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:29.045469    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6cf606-mjwrv"]
Nov 02 07:58:29.292514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:29.292474    4898 generic.go:356] "Generic (PLEG): container finished" podID="1ab85ca9-578d-4911-ab8f-89f8505ddd40" containerID="fc8b7d473013a5a3a7d55203c662c8c27f928226253c0dc172cffe73595c0799" exitCode=0
Nov 02 07:58:29.293093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:29.292567    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f" event={"ID":"1ab85ca9-578d-4911-ab8f-89f8505ddd40","Type":"ContainerDied","Data":"fc8b7d473013a5a3a7d55203c662c8c27f928226253c0dc172cffe73595c0799"}
Nov 02 07:58:29.470013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:29.469972    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d625389-08c3-4009-beb8-68cc7832967f" path="/var/lib/kubelet/pods/7d625389-08c3-4009-beb8-68cc7832967f/volumes"
Nov 02 07:58:30.229557 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.229527    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:58:30.273776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.273742    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:58:30.307952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.307875    4898 generic.go:356] "Generic (PLEG): container finished" podID="1409db03-535a-4343-902e-bcfcadd91650" containerID="819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b" exitCode=0
Nov 02 07:58:30.307952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.307920    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4" event={"ID":"1409db03-535a-4343-902e-bcfcadd91650","Type":"ContainerDied","Data":"819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b"}
Nov 02 07:58:30.308538 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.307974    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"
Nov 02 07:58:30.308538 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.307991    4898 scope.go:122] "RemoveContainer" containerID="819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b"
Nov 02 07:58:30.308538 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.307979    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4" event={"ID":"1409db03-535a-4343-902e-bcfcadd91650","Type":"ContainerDied","Data":"c897d66323875730fa166ed82beea530362ee3b6ddf7137bcd39234d605a1aa6"}
Nov 02 07:58:30.309987 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.309959    4898 generic.go:356] "Generic (PLEG): container finished" podID="1127b479-5f27-43d2-a86b-069a7cd84d39" containerID="ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97" exitCode=0
Nov 02 07:58:30.310109 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.310013    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"
Nov 02 07:58:30.310109 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.310045    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5" event={"ID":"1127b479-5f27-43d2-a86b-069a7cd84d39","Type":"ContainerDied","Data":"ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97"}
Nov 02 07:58:30.310109 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.310099    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5" event={"ID":"1127b479-5f27-43d2-a86b-069a7cd84d39","Type":"ContainerDied","Data":"3cd7dcb1f1095e825ecc1be8f3ce0a37efbd4244b7b37afeeedf5260c70148be"}
Nov 02 07:58:30.317155 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317129    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1127b479-5f27-43d2-a86b-069a7cd84d39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1127b479-5f27-43d2-a86b-069a7cd84d39-istiod-ca-cert\") pod \"1127b479-5f27-43d2-a86b-069a7cd84d39\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") "
Nov 02 07:58:30.317281 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317203    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-credential-socket\") pod \"1127b479-5f27-43d2-a86b-069a7cd84d39\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") "
Nov 02 07:58:30.317348 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317303    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1409db03-535a-4343-902e-bcfcadd91650-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1409db03-535a-4343-902e-bcfcadd91650-istio-podinfo\") pod \"1409db03-535a-4343-902e-bcfcadd91650\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") "
Nov 02 07:58:30.317404 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317378    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-data\") pod \"1409db03-535a-4343-902e-bcfcadd91650\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") "
Nov 02 07:58:30.317462 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317409    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-token\" (UniqueName: \"kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-token\") pod \"1127b479-5f27-43d2-a86b-069a7cd84d39\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") "
Nov 02 07:58:30.317462 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317440    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-kube-api-access-52wgq\" (UniqueName: \"kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-kube-api-access-52wgq\") pod \"1127b479-5f27-43d2-a86b-069a7cd84d39\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") "
Nov 02 07:58:30.317564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317478    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-certs\") pod \"1127b479-5f27-43d2-a86b-069a7cd84d39\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") "
Nov 02 07:58:30.317564 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317527    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-certs\") pod \"1409db03-535a-4343-902e-bcfcadd91650\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") "
Nov 02 07:58:30.317672 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317625    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-socket\") pod \"1409db03-535a-4343-902e-bcfcadd91650\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") "
Nov 02 07:58:30.317729 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317672    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-podinfo\") pod \"1127b479-5f27-43d2-a86b-069a7cd84d39\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") "
Nov 02 07:58:30.317729 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317709    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-data\") pod \"1127b479-5f27-43d2-a86b-069a7cd84d39\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") "
Nov 02 07:58:30.317836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317753    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-istio-token\" (UniqueName: \"kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-istio-token\") pod \"1409db03-535a-4343-902e-bcfcadd91650\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") "
Nov 02 07:58:30.317836 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317783    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-credential-socket\") pod \"1409db03-535a-4343-902e-bcfcadd91650\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") "
Nov 02 07:58:30.318108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317858    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-kube-api-access-smw2n\" (UniqueName: \"kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-kube-api-access-smw2n\") pod \"1409db03-535a-4343-902e-bcfcadd91650\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") "
Nov 02 07:58:30.318108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317909    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1409db03-535a-4343-902e-bcfcadd91650-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1409db03-535a-4343-902e-bcfcadd91650-istiod-ca-cert\") pod \"1409db03-535a-4343-902e-bcfcadd91650\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") "
Nov 02 07:58:30.318108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317951    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-socket\") pod \"1127b479-5f27-43d2-a86b-069a7cd84d39\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") "
Nov 02 07:58:30.318108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.317991    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-envoy\") pod \"1127b479-5f27-43d2-a86b-069a7cd84d39\" (UID: \"1127b479-5f27-43d2-a86b-069a7cd84d39\") "
Nov 02 07:58:30.318108 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.318065    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-envoy\") pod \"1409db03-535a-4343-902e-bcfcadd91650\" (UID: \"1409db03-535a-4343-902e-bcfcadd91650\") "
Nov 02 07:58:30.319022 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.318608    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-socket" pod "1409db03-535a-4343-902e-bcfcadd91650" (UID: "1409db03-535a-4343-902e-bcfcadd91650"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.319022 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.318790    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.319632 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.319356    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1127b479-5f27-43d2-a86b-069a7cd84d39-istiod-ca-cert" pod "1127b479-5f27-43d2-a86b-069a7cd84d39" (UID: "1127b479-5f27-43d2-a86b-069a7cd84d39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:58:30.320057 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.319950    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-credential-socket" pod "1409db03-535a-4343-902e-bcfcadd91650" (UID: "1409db03-535a-4343-902e-bcfcadd91650"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.320513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.320226    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-certs" pod "1127b479-5f27-43d2-a86b-069a7cd84d39" (UID: "1127b479-5f27-43d2-a86b-069a7cd84d39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.320513 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.320261    4898 scope.go:122] "RemoveContainer" containerID="819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b"
Nov 02 07:58:30.320678 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.320562    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-certs" pod "1409db03-535a-4343-902e-bcfcadd91650" (UID: "1409db03-535a-4343-902e-bcfcadd91650"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.320678 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:58:30.320661    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b\": container with ID starting with 819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b not found: ID does not exist" containerID="819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b"
Nov 02 07:58:30.320854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.320694    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b"} err="failed to get container status \"819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b\": rpc error: code = NotFound desc = could not find container \"819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b\": container with ID starting with 819afdf89ddd85a2559467d2af67b09950f1a86ee35ce7cf903c52b894279e2b not found: ID does not exist"
Nov 02 07:58:30.320854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.320718    4898 scope.go:122] "RemoveContainer" containerID="ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97"
Nov 02 07:58:30.321229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.321080    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-socket" pod "1127b479-5f27-43d2-a86b-069a7cd84d39" (UID: "1127b479-5f27-43d2-a86b-069a7cd84d39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.322184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.321350    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1409db03-535a-4343-902e-bcfcadd91650-istiod-ca-cert" pod "1409db03-535a-4343-902e-bcfcadd91650" (UID: "1409db03-535a-4343-902e-bcfcadd91650"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:58:30.322184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.321723    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-data" pod "1127b479-5f27-43d2-a86b-069a7cd84d39" (UID: "1127b479-5f27-43d2-a86b-069a7cd84d39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.322184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.321873    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-credential-socket" pod "1127b479-5f27-43d2-a86b-069a7cd84d39" (UID: "1127b479-5f27-43d2-a86b-069a7cd84d39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.322184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.322108    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-envoy" pod "1409db03-535a-4343-902e-bcfcadd91650" (UID: "1409db03-535a-4343-902e-bcfcadd91650"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.322674 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.322649    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-data" pod "1409db03-535a-4343-902e-bcfcadd91650" (UID: "1409db03-535a-4343-902e-bcfcadd91650"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.323097 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.323069    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-kube-api-access-52wgq" pod "1127b479-5f27-43d2-a86b-069a7cd84d39" (UID: "1127b479-5f27-43d2-a86b-069a7cd84d39"). InnerVolumeSpecName "kube-api-access-52wgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:30.324736 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.324703    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-envoy" pod "1127b479-5f27-43d2-a86b-069a7cd84d39" (UID: "1127b479-5f27-43d2-a86b-069a7cd84d39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:30.325074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.324958    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-kube-api-access-smw2n" pod "1409db03-535a-4343-902e-bcfcadd91650" (UID: "1409db03-535a-4343-902e-bcfcadd91650"). InnerVolumeSpecName "kube-api-access-smw2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:30.325074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.325042    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-istio-token" pod "1409db03-535a-4343-902e-bcfcadd91650" (UID: "1409db03-535a-4343-902e-bcfcadd91650"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:30.325229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.325082    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-token" pod "1127b479-5f27-43d2-a86b-069a7cd84d39" (UID: "1127b479-5f27-43d2-a86b-069a7cd84d39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:30.325229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.325086    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-podinfo" pod "1127b479-5f27-43d2-a86b-069a7cd84d39" (UID: "1127b479-5f27-43d2-a86b-069a7cd84d39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:58:30.326664 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.326632    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1409db03-535a-4343-902e-bcfcadd91650-istio-podinfo" pod "1409db03-535a-4343-902e-bcfcadd91650" (UID: "1409db03-535a-4343-902e-bcfcadd91650"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:58:30.368413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.368382    4898 scope.go:122] "RemoveContainer" containerID="ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97"
Nov 02 07:58:30.368974 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:58:30.368945    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97\": container with ID starting with ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97 not found: ID does not exist" containerID="ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97"
Nov 02 07:58:30.369135 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.368985    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97"} err="failed to get container status \"ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97\": rpc error: code = NotFound desc = could not find container \"ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97\": container with ID starting with ec8d1fd51cb5833403fad42c0e24c03aaa108fbb00b582f28935b0dd150c5d97 not found: ID does not exist"
Nov 02 07:58:30.420041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420009    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420041 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420041    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1127b479-5f27-43d2-a86b-069a7cd84d39-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420051    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420063    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1409db03-535a-4343-902e-bcfcadd91650-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420071    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420078    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420087    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52wgq\" (UniqueName: \"kubernetes.io/projected/1127b479-5f27-43d2-a86b-069a7cd84d39-kube-api-access-52wgq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420095    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420102    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420112    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420120    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420127    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420144    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1409db03-535a-4343-902e-bcfcadd91650-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420153    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smw2n\" (UniqueName: \"kubernetes.io/projected/1409db03-535a-4343-902e-bcfcadd91650-kube-api-access-smw2n\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420162    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1409db03-535a-4343-902e-bcfcadd91650-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420171    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.420252 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.420179    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1127b479-5f27-43d2-a86b-069a7cd84d39-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.522907 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.522865    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:30.627470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.625724    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-xks-ca\") pod \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\" (UID: \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\") "
Nov 02 07:58:30.627470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.625881    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-openshift-ca\") pod \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\" (UID: \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\") "
Nov 02 07:58:30.627470 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.626031    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab85ca9-578d-4911-ab8f-89f8505ddd40-kube-api-access-m2g8w\" (UniqueName: \"kubernetes.io/projected/1ab85ca9-578d-4911-ab8f-89f8505ddd40-kube-api-access-m2g8w\") pod \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\" (UID: \"1ab85ca9-578d-4911-ab8f-89f8505ddd40\") "
Nov 02 07:58:30.629400 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.629363    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab85ca9-578d-4911-ab8f-89f8505ddd40-kube-api-access-m2g8w" pod "1ab85ca9-578d-4911-ab8f-89f8505ddd40" (UID: "1ab85ca9-578d-4911-ab8f-89f8505ddd40"). InnerVolumeSpecName "kube-api-access-m2g8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:30.629631 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.629610    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-openshift-ca" pod "1ab85ca9-578d-4911-ab8f-89f8505ddd40" (UID: "1ab85ca9-578d-4911-ab8f-89f8505ddd40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:58:30.631994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.630449    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-xks-ca" pod "1ab85ca9-578d-4911-ab8f-89f8505ddd40" (UID: "1ab85ca9-578d-4911-ab8f-89f8505ddd40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:58:30.667866 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.667826    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"]
Nov 02 07:58:30.671594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.671568    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qmcj5"]
Nov 02 07:58:30.705321 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.705281    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"]
Nov 02 07:58:30.708961 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.708934    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-qzfk4"]
Nov 02 07:58:30.728225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.728184    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2g8w\" (UniqueName: \"kubernetes.io/projected/1ab85ca9-578d-4911-ab8f-89f8505ddd40-kube-api-access-m2g8w\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.728225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.728219    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:30.728225 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:30.728230    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab85ca9-578d-4911-ab8f-89f8505ddd40-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:31.321015 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:31.320883    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"
Nov 02 07:58:31.321465 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:31.320915    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f" event={"ID":"1ab85ca9-578d-4911-ab8f-89f8505ddd40","Type":"ContainerDied","Data":"e6cfe25fc45c99a9a654ed15623333318761e23e26043a4705aac487ee76ebb7"}
Nov 02 07:58:31.321465 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:31.321121    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6cfe25fc45c99a9a654ed15623333318761e23e26043a4705aac487ee76ebb7"
Nov 02 07:58:31.467120 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:31.467083    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1127b479-5f27-43d2-a86b-069a7cd84d39" path="/var/lib/kubelet/pods/1127b479-5f27-43d2-a86b-069a7cd84d39/volumes"
Nov 02 07:58:31.467654 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:31.467638    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1409db03-535a-4343-902e-bcfcadd91650" path="/var/lib/kubelet/pods/1409db03-535a-4343-902e-bcfcadd91650/volumes"
Nov 02 07:58:33.030255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:33.030216    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"]
Nov 02 07:58:33.033899 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:33.033864    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-693b3763-tj52l"]
Nov 02 07:58:33.471740 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:33.471707    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ace2b5a-db26-4bc6-9670-6a790d0f8b69" path="/var/lib/kubelet/pods/1ace2b5a-db26-4bc6-9670-6a790d0f8b69/volumes"
Nov 02 07:58:37.477831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.477800    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"]
Nov 02 07:58:37.478456 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.478440    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1409db03-535a-4343-902e-bcfcadd91650" containerName="istio-proxy"
Nov 02 07:58:37.478502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.478458    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1409db03-535a-4343-902e-bcfcadd91650" containerName="istio-proxy"
Nov 02 07:58:37.478502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.478479    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1ab85ca9-578d-4911-ab8f-89f8505ddd40" containerName="revoke-keys"
Nov 02 07:58:37.478502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.478485    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab85ca9-578d-4911-ab8f-89f8505ddd40" containerName="revoke-keys"
Nov 02 07:58:37.478605 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.478506    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1127b479-5f27-43d2-a86b-069a7cd84d39" containerName="istio-proxy"
Nov 02 07:58:37.478605 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.478510    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1127b479-5f27-43d2-a86b-069a7cd84d39" containerName="istio-proxy"
Nov 02 07:58:37.478605 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.478600    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1127b479-5f27-43d2-a86b-069a7cd84d39" containerName="istio-proxy"
Nov 02 07:58:37.478699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.478613    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1409db03-535a-4343-902e-bcfcadd91650" containerName="istio-proxy"
Nov 02 07:58:37.478699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.478622    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab85ca9-578d-4911-ab8f-89f8505ddd40" containerName="revoke-keys"
Nov 02 07:58:37.482197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.482180    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:37.504934 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.504880    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"]
Nov 02 07:58:37.521465 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.521409    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:58:37.521639 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.521527    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:58:37.521729 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.521641    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:58:37.521729 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.521720    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:58:37.537654 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.537620    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7njcz\" (UniqueName: \"kubernetes.io/projected/d76ec918-191f-46b5-ac33-98189801673a-kube-api-access-7njcz\") pod \"maas-api-revoke-sub-41e3a623-bzg7t\" (UID: \"d76ec918-191f-46b5-ac33-98189801673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:37.537968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.537946    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76ec918-191f-46b5-ac33-98189801673a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41e3a623-bzg7t\" (UID: \"d76ec918-191f-46b5-ac33-98189801673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:37.538277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.538247    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76ec918-191f-46b5-ac33-98189801673a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41e3a623-bzg7t\" (UID: \"d76ec918-191f-46b5-ac33-98189801673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:37.639909 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.639851    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76ec918-191f-46b5-ac33-98189801673a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41e3a623-bzg7t\" (UID: \"d76ec918-191f-46b5-ac33-98189801673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:37.640101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.640043    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76ec918-191f-46b5-ac33-98189801673a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41e3a623-bzg7t\" (UID: \"d76ec918-191f-46b5-ac33-98189801673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:37.640172 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.640155    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7njcz\" (UniqueName: \"kubernetes.io/projected/d76ec918-191f-46b5-ac33-98189801673a-kube-api-access-7njcz\") pod \"maas-api-revoke-sub-41e3a623-bzg7t\" (UID: \"d76ec918-191f-46b5-ac33-98189801673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:37.640666 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.640630    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76ec918-191f-46b5-ac33-98189801673a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41e3a623-bzg7t\" (UID: \"d76ec918-191f-46b5-ac33-98189801673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:37.642984 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.642953    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76ec918-191f-46b5-ac33-98189801673a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41e3a623-bzg7t\" (UID: \"d76ec918-191f-46b5-ac33-98189801673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:37.664750 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.664719    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7njcz\" (UniqueName: \"kubernetes.io/projected/d76ec918-191f-46b5-ac33-98189801673a-kube-api-access-7njcz\") pod \"maas-api-revoke-sub-41e3a623-bzg7t\" (UID: \"d76ec918-191f-46b5-ac33-98189801673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:37.797052 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:37.796963    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:38.102127 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:58:38.102032    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76ec918_191f_46b5_ac33_98189801673a.slice/crio-00f407d13c20412a6e4d87f7f6edcefcce339f4b21f4be00f34972debd58f877 WatchSource:0}: Error finding container 00f407d13c20412a6e4d87f7f6edcefcce339f4b21f4be00f34972debd58f877: Status 404 returned error can't find the container with id 00f407d13c20412a6e4d87f7f6edcefcce339f4b21f4be00f34972debd58f877
Nov 02 07:58:38.104497 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:38.104471    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"]
Nov 02 07:58:38.433369 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:38.433327    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t" event={"ID":"d76ec918-191f-46b5-ac33-98189801673a","Type":"ContainerStarted","Data":"3f75ba4597a65fef8b75e43e6f0d34315628815a73e05cc091693a4d678f5cf0"}
Nov 02 07:58:38.433626 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:38.433403    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t" event={"ID":"d76ec918-191f-46b5-ac33-98189801673a","Type":"ContainerStarted","Data":"00f407d13c20412a6e4d87f7f6edcefcce339f4b21f4be00f34972debd58f877"}
Nov 02 07:58:38.481227 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:38.481163    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t" podStartSLOduration=1.481150019 podStartE2EDuration="1.481150019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:58:38.480653415 +0000 UTC m=+2069.170980385" watchObservedRunningTime="2026-11-02 07:58:38.481150019 +0000 UTC m=+2069.171476985"
Nov 02 07:58:38.951383 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:38.951339    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"]
Nov 02 07:58:39.443178 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:39.443143    4898 generic.go:356] "Generic (PLEG): container finished" podID="d76ec918-191f-46b5-ac33-98189801673a" containerID="3f75ba4597a65fef8b75e43e6f0d34315628815a73e05cc091693a4d678f5cf0" exitCode=0
Nov 02 07:58:39.443387 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:39.443227    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t" event={"ID":"d76ec918-191f-46b5-ac33-98189801673a","Type":"ContainerDied","Data":"3f75ba4597a65fef8b75e43e6f0d34315628815a73e05cc091693a4d678f5cf0"}
Nov 02 07:58:40.623715 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:40.623680    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:40.706413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:40.706313    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76ec918-191f-46b5-ac33-98189801673a-kube-api-access-7njcz\" (UniqueName: \"kubernetes.io/projected/d76ec918-191f-46b5-ac33-98189801673a-kube-api-access-7njcz\") pod \"d76ec918-191f-46b5-ac33-98189801673a\" (UID: \"d76ec918-191f-46b5-ac33-98189801673a\") "
Nov 02 07:58:40.706413 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:40.706409    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d76ec918-191f-46b5-ac33-98189801673a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76ec918-191f-46b5-ac33-98189801673a-maas-api-openshift-ca\") pod \"d76ec918-191f-46b5-ac33-98189801673a\" (UID: \"d76ec918-191f-46b5-ac33-98189801673a\") "
Nov 02 07:58:40.706632 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:40.706559    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d76ec918-191f-46b5-ac33-98189801673a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76ec918-191f-46b5-ac33-98189801673a-maas-api-xks-ca\") pod \"d76ec918-191f-46b5-ac33-98189801673a\" (UID: \"d76ec918-191f-46b5-ac33-98189801673a\") "
Nov 02 07:58:40.706875 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:40.706835    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76ec918-191f-46b5-ac33-98189801673a-maas-api-openshift-ca" pod "d76ec918-191f-46b5-ac33-98189801673a" (UID: "d76ec918-191f-46b5-ac33-98189801673a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:58:40.707194 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:40.707172    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d76ec918-191f-46b5-ac33-98189801673a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:40.709197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:40.709163    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d76ec918-191f-46b5-ac33-98189801673a-maas-api-xks-ca" pod "d76ec918-191f-46b5-ac33-98189801673a" (UID: "d76ec918-191f-46b5-ac33-98189801673a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:58:40.711125 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:40.711102    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76ec918-191f-46b5-ac33-98189801673a-kube-api-access-7njcz" pod "d76ec918-191f-46b5-ac33-98189801673a" (UID: "d76ec918-191f-46b5-ac33-98189801673a"). InnerVolumeSpecName "kube-api-access-7njcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:40.808952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:40.808916    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7njcz\" (UniqueName: \"kubernetes.io/projected/d76ec918-191f-46b5-ac33-98189801673a-kube-api-access-7njcz\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:40.808952 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:40.808948    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d76ec918-191f-46b5-ac33-98189801673a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:41.462808 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:41.462778    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"
Nov 02 07:58:41.467044 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:41.467016    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t" event={"ID":"d76ec918-191f-46b5-ac33-98189801673a","Type":"ContainerDied","Data":"00f407d13c20412a6e4d87f7f6edcefcce339f4b21f4be00f34972debd58f877"}
Nov 02 07:58:41.467188 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:41.467064    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00f407d13c20412a6e4d87f7f6edcefcce339f4b21f4be00f34972debd58f877"
Nov 02 07:58:44.989415 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:58:44.989375    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:58:53.976906 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:53.976840    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s" podUID="0389b0c4-5ad0-4a32-982c-e36ca577f3ac" containerName="main" containerID="cri-o://7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694" gracePeriod=45
Nov 02 07:58:54.321360 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.321331    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:54.441302 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.441265    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tls-certs\") pod \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") "
Nov 02 07:58:54.441519 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.441458    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tmp-dir\") pod \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") "
Nov 02 07:58:54.441630 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.441607    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-dshm\") pod \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") "
Nov 02 07:58:54.441706 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.441659    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-kube-api-access-gbfmg\" (UniqueName: \"kubernetes.io/projected/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-kube-api-access-gbfmg\") pod \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") "
Nov 02 07:58:54.441762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.441702    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-model-cache\") pod \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") "
Nov 02 07:58:54.441762 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.441741    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-home\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-home\") pod \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\" (UID: \"0389b0c4-5ad0-4a32-982c-e36ca577f3ac\") "
Nov 02 07:58:54.442710 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.442685    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-home" pod "0389b0c4-5ad0-4a32-982c-e36ca577f3ac" (UID: "0389b0c4-5ad0-4a32-982c-e36ca577f3ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:54.443073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.443052    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-model-cache" pod "0389b0c4-5ad0-4a32-982c-e36ca577f3ac" (UID: "0389b0c4-5ad0-4a32-982c-e36ca577f3ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:54.446176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.444739    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tmp-dir" pod "0389b0c4-5ad0-4a32-982c-e36ca577f3ac" (UID: "0389b0c4-5ad0-4a32-982c-e36ca577f3ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:54.448268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.448213    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tls-certs" pod "0389b0c4-5ad0-4a32-982c-e36ca577f3ac" (UID: "0389b0c4-5ad0-4a32-982c-e36ca577f3ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:58:54.448268 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.448208    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-dshm" pod "0389b0c4-5ad0-4a32-982c-e36ca577f3ac" (UID: "0389b0c4-5ad0-4a32-982c-e36ca577f3ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:58:54.450469 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.450436    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-kube-api-access-gbfmg" pod "0389b0c4-5ad0-4a32-982c-e36ca577f3ac" (UID: "0389b0c4-5ad0-4a32-982c-e36ca577f3ac"). InnerVolumeSpecName "kube-api-access-gbfmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:54.582648 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.574244    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:54.584503 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.584478    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:54.584687 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.584671    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbfmg\" (UniqueName: \"kubernetes.io/projected/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-kube-api-access-gbfmg\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:54.584790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.584778    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:54.584881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.584871    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:54.585012 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.584995    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0389b0c4-5ad0-4a32-982c-e36ca577f3ac-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:54.637463 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.637428    4898 generic.go:356] "Generic (PLEG): container finished" podID="0389b0c4-5ad0-4a32-982c-e36ca577f3ac" containerID="7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694" exitCode=0
Nov 02 07:58:54.637699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.637467    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"
Nov 02 07:58:54.637699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.637509    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s" event={"ID":"0389b0c4-5ad0-4a32-982c-e36ca577f3ac","Type":"ContainerDied","Data":"7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694"}
Nov 02 07:58:54.637699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.637554    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s" event={"ID":"0389b0c4-5ad0-4a32-982c-e36ca577f3ac","Type":"ContainerDied","Data":"2abc9e42536523814b478240cd81b15bd184bc060e21464682dd37205b07a508"}
Nov 02 07:58:54.637699 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.637572    4898 scope.go:122] "RemoveContainer" containerID="7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694"
Nov 02 07:58:54.649184 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.649167    4898 scope.go:122] "RemoveContainer" containerID="7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694"
Nov 02 07:58:54.649474 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:58:54.649455    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694\": container with ID starting with 7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694 not found: ID does not exist" containerID="7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694"
Nov 02 07:58:54.649538 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.649480    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694"} err="failed to get container status \"7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694\": rpc error: code = NotFound desc = could not find container \"7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694\": container with ID starting with 7908e21f4fa50e74a08ebccaa59d1d0361737fc46b52118693a1776614c99694 not found: ID does not exist"
Nov 02 07:58:54.692144 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.692100    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"]
Nov 02 07:58:54.706290 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:54.706264    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s"]
Nov 02 07:58:55.170986 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:55.170940    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-ac716b/explicit-model-ac716b-kserve-c665dbfdc-cdd4s" podUID="0389b0c4-5ad0-4a32-982c-e36ca577f3ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 07:58:55.466213 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:55.466133    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0389b0c4-5ad0-4a32-982c-e36ca577f3ac" path="/var/lib/kubelet/pods/0389b0c4-5ad0-4a32-982c-e36ca577f3ac/volumes"
Nov 02 07:58:56.352486 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.352087    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"]
Nov 02 07:58:56.353191 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.353171    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0389b0c4-5ad0-4a32-982c-e36ca577f3ac" containerName="main"
Nov 02 07:58:56.353275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.353193    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0389b0c4-5ad0-4a32-982c-e36ca577f3ac" containerName="main"
Nov 02 07:58:56.353275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.353218    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d76ec918-191f-46b5-ac33-98189801673a" containerName="revoke-keys"
Nov 02 07:58:56.353275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.353226    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76ec918-191f-46b5-ac33-98189801673a" containerName="revoke-keys"
Nov 02 07:58:56.353404 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.353370    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76ec918-191f-46b5-ac33-98189801673a" containerName="revoke-keys"
Nov 02 07:58:56.353404 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.353395    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0389b0c4-5ad0-4a32-982c-e36ca577f3ac" containerName="main"
Nov 02 07:58:56.359667 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.359643    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:58:56.381832 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.381799    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"]
Nov 02 07:58:56.391173 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.391144    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:58:56.391340 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.391289    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:58:56.391421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.391340    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:58:56.391421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.391399    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:58:56.512433 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.512382    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc8w8\" (UniqueName: \"kubernetes.io/projected/007bb236-057e-450e-85ef-a49291f5a91c-kube-api-access-kc8w8\") pod \"maas-api-revoke-sub-395b6f0b-p7ftm\" (UID: \"007bb236-057e-450e-85ef-a49291f5a91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:58:56.512615 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.512583    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-395b6f0b-p7ftm\" (UID: \"007bb236-057e-450e-85ef-a49291f5a91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:58:56.512779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.512756    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-395b6f0b-p7ftm\" (UID: \"007bb236-057e-450e-85ef-a49291f5a91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:58:56.613692 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.613604    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-395b6f0b-p7ftm\" (UID: \"007bb236-057e-450e-85ef-a49291f5a91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:58:56.613692 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.613676    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc8w8\" (UniqueName: \"kubernetes.io/projected/007bb236-057e-450e-85ef-a49291f5a91c-kube-api-access-kc8w8\") pod \"maas-api-revoke-sub-395b6f0b-p7ftm\" (UID: \"007bb236-057e-450e-85ef-a49291f5a91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:58:56.614011 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.613785    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-395b6f0b-p7ftm\" (UID: \"007bb236-057e-450e-85ef-a49291f5a91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:58:56.614509 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.614484    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-395b6f0b-p7ftm\" (UID: \"007bb236-057e-450e-85ef-a49291f5a91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:58:56.616406 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.616387    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-395b6f0b-p7ftm\" (UID: \"007bb236-057e-450e-85ef-a49291f5a91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:58:56.637859 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.637655    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc8w8\" (UniqueName: \"kubernetes.io/projected/007bb236-057e-450e-85ef-a49291f5a91c-kube-api-access-kc8w8\") pod \"maas-api-revoke-sub-395b6f0b-p7ftm\" (UID: \"007bb236-057e-450e-85ef-a49291f5a91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:58:56.685959 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:56.685921    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:58:57.000781 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:58:57.000750    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007bb236_057e_450e_85ef_a49291f5a91c.slice/crio-e9e777047a2548bf616234cad94167c403395f8406e06d4095459f6b367a4756 WatchSource:0}: Error finding container e9e777047a2548bf616234cad94167c403395f8406e06d4095459f6b367a4756: Status 404 returned error can't find the container with id e9e777047a2548bf616234cad94167c403395f8406e06d4095459f6b367a4756
Nov 02 07:58:57.002962 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:57.002939    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"]
Nov 02 07:58:57.666717 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:57.666678    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm" event={"ID":"007bb236-057e-450e-85ef-a49291f5a91c","Type":"ContainerStarted","Data":"bb904bc56d309f027c9eef3edc45694447b6ab4e031771409a0b6427e0621f1c"}
Nov 02 07:58:57.667142 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:57.666742    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm" event={"ID":"007bb236-057e-450e-85ef-a49291f5a91c","Type":"ContainerStarted","Data":"e9e777047a2548bf616234cad94167c403395f8406e06d4095459f6b367a4756"}
Nov 02 07:58:57.707207 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:57.707152    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm" podStartSLOduration=1.707137938 podStartE2EDuration="1.707137938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:58:57.706221862 +0000 UTC m=+2088.396548821" watchObservedRunningTime="2026-11-02 07:58:57.707137938 +0000 UTC m=+2088.397464901"
Nov 02 07:58:58.677531 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:58.677492    4898 generic.go:356] "Generic (PLEG): container finished" podID="007bb236-057e-450e-85ef-a49291f5a91c" containerID="bb904bc56d309f027c9eef3edc45694447b6ab4e031771409a0b6427e0621f1c" exitCode=0
Nov 02 07:58:58.677995 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:58.677582    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm" event={"ID":"007bb236-057e-450e-85ef-a49291f5a91c","Type":"ContainerDied","Data":"bb904bc56d309f027c9eef3edc45694447b6ab4e031771409a0b6427e0621f1c"}
Nov 02 07:58:59.855703 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:59.855676    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:58:59.961264 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:59.961175    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007bb236-057e-450e-85ef-a49291f5a91c-kube-api-access-kc8w8\" (UniqueName: \"kubernetes.io/projected/007bb236-057e-450e-85ef-a49291f5a91c-kube-api-access-kc8w8\") pod \"007bb236-057e-450e-85ef-a49291f5a91c\" (UID: \"007bb236-057e-450e-85ef-a49291f5a91c\") "
Nov 02 07:58:59.961453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:59.961259    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-openshift-ca\") pod \"007bb236-057e-450e-85ef-a49291f5a91c\" (UID: \"007bb236-057e-450e-85ef-a49291f5a91c\") "
Nov 02 07:58:59.961453 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:59.961402    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-xks-ca\") pod \"007bb236-057e-450e-85ef-a49291f5a91c\" (UID: \"007bb236-057e-450e-85ef-a49291f5a91c\") "
Nov 02 07:58:59.965095 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:59.965062    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-openshift-ca" pod "007bb236-057e-450e-85ef-a49291f5a91c" (UID: "007bb236-057e-450e-85ef-a49291f5a91c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:58:59.965461 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:59.965439    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:58:59.968162 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:59.968101    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007bb236-057e-450e-85ef-a49291f5a91c-kube-api-access-kc8w8" pod "007bb236-057e-450e-85ef-a49291f5a91c" (UID: "007bb236-057e-450e-85ef-a49291f5a91c"). InnerVolumeSpecName "kube-api-access-kc8w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:58:59.969332 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:58:59.969308    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-xks-ca" pod "007bb236-057e-450e-85ef-a49291f5a91c" (UID: "007bb236-057e-450e-85ef-a49291f5a91c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:59:00.067511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:00.067476    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007bb236-057e-450e-85ef-a49291f5a91c-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:00.067511 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:00.067509    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc8w8\" (UniqueName: \"kubernetes.io/projected/007bb236-057e-450e-85ef-a49291f5a91c-kube-api-access-kc8w8\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:00.695788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:00.695757    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"
Nov 02 07:59:00.695980 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:00.695792    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm" event={"ID":"007bb236-057e-450e-85ef-a49291f5a91c","Type":"ContainerDied","Data":"e9e777047a2548bf616234cad94167c403395f8406e06d4095459f6b367a4756"}
Nov 02 07:59:00.695980 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:00.695844    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e777047a2548bf616234cad94167c403395f8406e06d4095459f6b367a4756"
Nov 02 07:59:06.728116 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.728086    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"]
Nov 02 07:59:06.728744 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.728727    4898 cpu_manager.go:414] "containerMap: removing container" podUID="007bb236-057e-450e-85ef-a49291f5a91c" containerName="revoke-keys"
Nov 02 07:59:06.728744 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.728744    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007bb236-057e-450e-85ef-a49291f5a91c" containerName="revoke-keys"
Nov 02 07:59:06.728931 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.728876    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="007bb236-057e-450e-85ef-a49291f5a91c" containerName="revoke-keys"
Nov 02 07:59:06.732954 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.732929    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:06.754834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.754806    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fbb756d-jgdnk\" (UID: \"4517047f-61ba-4cf9-a710-36e767dc7038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:06.755054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.755032    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtfg9\" (UniqueName: \"kubernetes.io/projected/4517047f-61ba-4cf9-a710-36e767dc7038-kube-api-access-jtfg9\") pod \"maas-api-revoke-sub-3fbb756d-jgdnk\" (UID: \"4517047f-61ba-4cf9-a710-36e767dc7038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:06.755253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.755231    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fbb756d-jgdnk\" (UID: \"4517047f-61ba-4cf9-a710-36e767dc7038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:06.755973 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.755641    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"]
Nov 02 07:59:06.762192 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.762169    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:59:06.763054 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.762374    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:59:06.763358 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.762510    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:59:06.764085 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.764066    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:59:06.856533 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.856496    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fbb756d-jgdnk\" (UID: \"4517047f-61ba-4cf9-a710-36e767dc7038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:06.856774 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.856757    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fbb756d-jgdnk\" (UID: \"4517047f-61ba-4cf9-a710-36e767dc7038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:06.856859 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.856797    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtfg9\" (UniqueName: \"kubernetes.io/projected/4517047f-61ba-4cf9-a710-36e767dc7038-kube-api-access-jtfg9\") pod \"maas-api-revoke-sub-3fbb756d-jgdnk\" (UID: \"4517047f-61ba-4cf9-a710-36e767dc7038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:06.857597 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.857559    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fbb756d-jgdnk\" (UID: \"4517047f-61ba-4cf9-a710-36e767dc7038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:06.859150 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.859124    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fbb756d-jgdnk\" (UID: \"4517047f-61ba-4cf9-a710-36e767dc7038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:06.880766 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:06.880736    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtfg9\" (UniqueName: \"kubernetes.io/projected/4517047f-61ba-4cf9-a710-36e767dc7038-kube-api-access-jtfg9\") pod \"maas-api-revoke-sub-3fbb756d-jgdnk\" (UID: \"4517047f-61ba-4cf9-a710-36e767dc7038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:07.054881 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:07.054803    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:07.361774 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:59:07.361736    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4517047f_61ba_4cf9_a710_36e767dc7038.slice/crio-cb2b4c49c84b43942535610ecfb443cabb443e0c9a46a5f94488e151d2529b77 WatchSource:0}: Error finding container cb2b4c49c84b43942535610ecfb443cabb443e0c9a46a5f94488e151d2529b77: Status 404 returned error can't find the container with id cb2b4c49c84b43942535610ecfb443cabb443e0c9a46a5f94488e151d2529b77
Nov 02 07:59:07.363165 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:07.363135    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"]
Nov 02 07:59:07.786441 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:07.786392    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk" event={"ID":"4517047f-61ba-4cf9-a710-36e767dc7038","Type":"ContainerStarted","Data":"64ca3b044bdcb97a94cbbf1997364304b4438b743c88439f11fc21d2bbf75135"}
Nov 02 07:59:07.787051 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:07.787016    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk" event={"ID":"4517047f-61ba-4cf9-a710-36e767dc7038","Type":"ContainerStarted","Data":"cb2b4c49c84b43942535610ecfb443cabb443e0c9a46a5f94488e151d2529b77"}
Nov 02 07:59:07.829514 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:07.829454    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk" podStartSLOduration=1.829439043 podStartE2EDuration="1.829439043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:07.827806373 +0000 UTC m=+2098.518133336" watchObservedRunningTime="2026-11-02 07:59:07.829439043 +0000 UTC m=+2098.519766024"
Nov 02 07:59:08.796782 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:08.796746    4898 generic.go:356] "Generic (PLEG): container finished" podID="4517047f-61ba-4cf9-a710-36e767dc7038" containerID="64ca3b044bdcb97a94cbbf1997364304b4438b743c88439f11fc21d2bbf75135" exitCode=0
Nov 02 07:59:08.797229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:08.796833    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk" event={"ID":"4517047f-61ba-4cf9-a710-36e767dc7038","Type":"ContainerDied","Data":"64ca3b044bdcb97a94cbbf1997364304b4438b743c88439f11fc21d2bbf75135"}
Nov 02 07:59:09.971289 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:09.971262    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:10.002587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.002555    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4517047f-61ba-4cf9-a710-36e767dc7038-kube-api-access-jtfg9\" (UniqueName: \"kubernetes.io/projected/4517047f-61ba-4cf9-a710-36e767dc7038-kube-api-access-jtfg9\") pod \"4517047f-61ba-4cf9-a710-36e767dc7038\" (UID: \"4517047f-61ba-4cf9-a710-36e767dc7038\") "
Nov 02 07:59:10.002816 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.002793    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-xks-ca\") pod \"4517047f-61ba-4cf9-a710-36e767dc7038\" (UID: \"4517047f-61ba-4cf9-a710-36e767dc7038\") "
Nov 02 07:59:10.003039 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.003013    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-openshift-ca\") pod \"4517047f-61ba-4cf9-a710-36e767dc7038\" (UID: \"4517047f-61ba-4cf9-a710-36e767dc7038\") "
Nov 02 07:59:10.003439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.003411    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-openshift-ca" pod "4517047f-61ba-4cf9-a710-36e767dc7038" (UID: "4517047f-61ba-4cf9-a710-36e767dc7038"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:10.003655 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.003635    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:10.004927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.004898    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4517047f-61ba-4cf9-a710-36e767dc7038-kube-api-access-jtfg9" pod "4517047f-61ba-4cf9-a710-36e767dc7038" (UID: "4517047f-61ba-4cf9-a710-36e767dc7038"). InnerVolumeSpecName "kube-api-access-jtfg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:10.005025 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.004940    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-xks-ca" pod "4517047f-61ba-4cf9-a710-36e767dc7038" (UID: "4517047f-61ba-4cf9-a710-36e767dc7038"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:59:10.104696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.104620    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4517047f-61ba-4cf9-a710-36e767dc7038-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:10.104696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.104653    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtfg9\" (UniqueName: \"kubernetes.io/projected/4517047f-61ba-4cf9-a710-36e767dc7038-kube-api-access-jtfg9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:10.241521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.241463    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:59:10.243063 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.243035    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 07:59:10.816243 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.816208    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"
Nov 02 07:59:10.817505 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.816313    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk" event={"ID":"4517047f-61ba-4cf9-a710-36e767dc7038","Type":"ContainerDied","Data":"cb2b4c49c84b43942535610ecfb443cabb443e0c9a46a5f94488e151d2529b77"}
Nov 02 07:59:10.817635 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.817550    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb2b4c49c84b43942535610ecfb443cabb443e0c9a46a5f94488e151d2529b77"
Nov 02 07:59:10.839285 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.839255    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"]
Nov 02 07:59:10.840305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.840279    4898 cpu_manager.go:414] "containerMap: removing container" podUID="4517047f-61ba-4cf9-a710-36e767dc7038" containerName="revoke-keys"
Nov 02 07:59:10.840305 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.840304    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4517047f-61ba-4cf9-a710-36e767dc7038" containerName="revoke-keys"
Nov 02 07:59:10.841449 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.840548    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4517047f-61ba-4cf9-a710-36e767dc7038" containerName="revoke-keys"
Nov 02 07:59:10.851341 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.851313    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:10.866815 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.866790    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"]
Nov 02 07:59:10.878872 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.878850    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c712b1-openshift-default-dockercfg-tlxs4\""
Nov 02 07:59:10.917914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.916386    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv9t8\" (UniqueName: \"kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-kube-api-access-bv9t8\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:10.917914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.916519    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-data\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:10.917914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.916592    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-token\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:10.917914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.916689    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-socket\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:10.917914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.916768    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-podinfo\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:10.917914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.916959    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istiod-ca-cert\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:10.917914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.917161    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-credential-socket\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:10.917914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.917238    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-certs\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:10.917914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:10.917286    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-envoy\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.037913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.031723    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-certs\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.037913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.031776    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-envoy\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.037913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.031809    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv9t8\" (UniqueName: \"kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-kube-api-access-bv9t8\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.037913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.031902    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-data\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.037913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.031948    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-token\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.037913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.031996    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-socket\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.037913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.032032    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-podinfo\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.037913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.032164    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istiod-ca-cert\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.037913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.032274    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-credential-socket\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.037913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.032742    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-credential-socket\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.037913 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.033004    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-certs\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.056927 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.041778    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-data\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.057466 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.053742    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-socket\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.057639 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.054788    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istiod-ca-cert\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.057746 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.057397    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-envoy\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.067078 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.067003    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-podinfo\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.088006 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.087931    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-token\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.116914 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.109937    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv9t8\" (UniqueName: \"kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-kube-api-access-bv9t8\") pod \"e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") " pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.185908 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.185857    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:11.582779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.582727    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"]
Nov 02 07:59:11.586994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.586225    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:11.586994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.586310    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:11.586994 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.586351    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:11.830343 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.830244    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" event={"ID":"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09","Type":"ContainerStarted","Data":"8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456"}
Nov 02 07:59:11.830343 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.830315    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" event={"ID":"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09","Type":"ContainerStarted","Data":"06378a476f1a7507282b3acf5ffa963a05761c7110d458c5634b8764fe5bfe75"}
Nov 02 07:59:11.880340 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.880277    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" podStartSLOduration=1.88025743 podStartE2EDuration="1.88025743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:11.878750596 +0000 UTC m=+2102.569077599" watchObservedRunningTime="2026-11-02 07:59:11.88025743 +0000 UTC m=+2102.570584395"
Nov 02 07:59:11.983357 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:11.983326    4898 scope.go:122] "RemoveContainer" containerID="ec8b385745bead599f0c98172d2114d38059e9704071846cc76f12f896a5f787"
Nov 02 07:59:12.049438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:12.049238    4898 scope.go:122] "RemoveContainer" containerID="0495c2368cfdd09ceae43d16a9d4425dedb0dc77c965872d3f2851da9d919590"
Nov 02 07:59:12.071751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:12.071727    4898 scope.go:122] "RemoveContainer" containerID="e691c6d92d383a65221dc7134357631daa7ed5a1c7824b7de6ea2182df95ef50"
Nov 02 07:59:12.082454 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:12.082431    4898 scope.go:122] "RemoveContainer" containerID="46b80982b3a1fad95b1b8d76b3bb1a508347975069d780a892006e4cd15299fc"
Nov 02 07:59:12.091776 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:12.091756    4898 scope.go:122] "RemoveContainer" containerID="ecd526c55205128f6540d22ab63fe68c210a6da2f703d62b5d8d8620eb93368c"
Nov 02 07:59:12.105028 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:12.105004    4898 scope.go:122] "RemoveContainer" containerID="a655d993d1b3bf7f40250bfa9966d845b977c85ecf83e2c32937411fc3fb422a"
Nov 02 07:59:12.186879 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:12.186852    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:12.192315 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:12.192291    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:12.840187 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:12.840156    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:12.841162 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:12.841141    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 07:59:14.990299 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:59:14.990262    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:59:21.335906 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:21.335860    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"]
Nov 02 07:59:21.367081 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:21.367052    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"]
Nov 02 07:59:22.191813 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.191728    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"]
Nov 02 07:59:22.197362 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.197335    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.226571 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.226543    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"]
Nov 02 07:59:22.226790 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.226769    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-696c7a-openshift-default-dockercfg-2kkh2\""
Nov 02 07:59:22.235671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.235229    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-data\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.235671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.235288    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9609fcde-fbb5-409b-82bf-78eaec870453-istio-podinfo\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.235671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.235332    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9609fcde-fbb5-409b-82bf-78eaec870453-istiod-ca-cert\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.235671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.235366    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-credential-socket\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.235671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.235481    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-envoy\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.235671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.235549    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-certs\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.235671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.235620    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nn7l\" (UniqueName: \"kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-kube-api-access-2nn7l\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.235671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.235654    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-socket\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.235671 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.235677    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-istio-token\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.336645 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.336602    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-envoy\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.336701    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-certs\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.336781    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nn7l\" (UniqueName: \"kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-kube-api-access-2nn7l\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.336823    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-socket\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.336847    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-istio-token\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.336915    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-data\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.336967    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9609fcde-fbb5-409b-82bf-78eaec870453-istio-podinfo\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.337010    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9609fcde-fbb5-409b-82bf-78eaec870453-istiod-ca-cert\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.337056    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-credential-socket\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337556 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.337233    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-certs\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337556 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.337323    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-data\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337556 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.337459    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-socket\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337556 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.337507    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-credential-socket\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.337870 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.337846    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9609fcde-fbb5-409b-82bf-78eaec870453-istiod-ca-cert\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.339783 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.339757    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9609fcde-fbb5-409b-82bf-78eaec870453-istio-podinfo\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.340074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.340054    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-envoy\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.361026 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.360995    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nn7l\" (UniqueName: \"kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-kube-api-access-2nn7l\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.361536 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.361512    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-istio-token\") pod \"e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") " pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.536982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.529170    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:22.572637 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.572575    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"]
Nov 02 07:59:22.572905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.572838    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" podUID="534095d2-847d-4712-8f8d-acb19d774364" containerName="istio-proxy" containerID="cri-o://03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f" gracePeriod=30
Nov 02 07:59:22.959264 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:59:22.959232    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9609fcde_fbb5_409b_82bf_78eaec870453.slice/crio-9901175637f9914a9eaabaefcb1ccb7f2d3257799474dbcd35f1729aff82cfa6 WatchSource:0}: Error finding container 9901175637f9914a9eaabaefcb1ccb7f2d3257799474dbcd35f1729aff82cfa6: Status 404 returned error can't find the container with id 9901175637f9914a9eaabaefcb1ccb7f2d3257799474dbcd35f1729aff82cfa6
Nov 02 07:59:22.961286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.961261    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"]
Nov 02 07:59:22.962160 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.962128    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:22.962218 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.962196    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:22.962255 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:22.962225    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 07:59:23.930905 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:23.930854    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" event={"ID":"9609fcde-fbb5-409b-82bf-78eaec870453","Type":"ContainerStarted","Data":"9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e"}
Nov 02 07:59:23.931306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:23.930971    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" event={"ID":"9609fcde-fbb5-409b-82bf-78eaec870453","Type":"ContainerStarted","Data":"9901175637f9914a9eaabaefcb1ccb7f2d3257799474dbcd35f1729aff82cfa6"}
Nov 02 07:59:23.989806 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:23.989753    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" podStartSLOduration=1.98973981 podStartE2EDuration="1.98973981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:23.988040216 +0000 UTC m=+2114.678367181" watchObservedRunningTime="2026-11-02 07:59:23.98973981 +0000 UTC m=+2114.680066774"
Nov 02 07:59:24.535382 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:24.535343    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:24.541421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:24.541391    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:24.938986 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:24.938951    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:24.939969 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:24.939945    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 07:59:26.049067 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.049037    4898 patch_prober.go:29] interesting pod/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 07:59:26.049464 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.049089    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" podUID="534095d2-847d-4712-8f8d-acb19d774364" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 07:59:26.336521 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.336423    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l" podUID="ff6ee21d-7956-4d2c-b00a-8e38afe9c585" containerName="payload-processing" containerID="cri-o://963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859" gracePeriod=25
Nov 02 07:59:26.367338 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.367293    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf" podUID="969f6f63-470f-4b22-b80b-cc3f33fb6f14" containerName="payload-pre-processing" containerID="cri-o://0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7" gracePeriod=25
Nov 02 07:59:26.694922 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.694877    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"
Nov 02 07:59:26.723498 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.723475    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"
Nov 02 07:59:26.843528 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.843487    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969f6f63-470f-4b22-b80b-cc3f33fb6f14-kube-api-access-tl8rx\" (UniqueName: \"kubernetes.io/projected/969f6f63-470f-4b22-b80b-cc3f33fb6f14-kube-api-access-tl8rx\") pod \"969f6f63-470f-4b22-b80b-cc3f33fb6f14\" (UID: \"969f6f63-470f-4b22-b80b-cc3f33fb6f14\") "
Nov 02 07:59:26.843791 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.843760    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-kube-api-access-g967m\" (UniqueName: \"kubernetes.io/projected/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-kube-api-access-g967m\") pod \"ff6ee21d-7956-4d2c-b00a-8e38afe9c585\" (UID: \"ff6ee21d-7956-4d2c-b00a-8e38afe9c585\") "
Nov 02 07:59:26.843876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.843818    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-plugins-config-volume\") pod \"ff6ee21d-7956-4d2c-b00a-8e38afe9c585\" (UID: \"ff6ee21d-7956-4d2c-b00a-8e38afe9c585\") "
Nov 02 07:59:26.844065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.843955    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/969f6f63-470f-4b22-b80b-cc3f33fb6f14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969f6f63-470f-4b22-b80b-cc3f33fb6f14-plugins-config-volume\") pod \"969f6f63-470f-4b22-b80b-cc3f33fb6f14\" (UID: \"969f6f63-470f-4b22-b80b-cc3f33fb6f14\") "
Nov 02 07:59:26.845095 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.845064    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969f6f63-470f-4b22-b80b-cc3f33fb6f14-plugins-config-volume" pod "969f6f63-470f-4b22-b80b-cc3f33fb6f14" (UID: "969f6f63-470f-4b22-b80b-cc3f33fb6f14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:26.847495 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.847467    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-plugins-config-volume" pod "ff6ee21d-7956-4d2c-b00a-8e38afe9c585" (UID: "ff6ee21d-7956-4d2c-b00a-8e38afe9c585"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:26.849115 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.849082    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969f6f63-470f-4b22-b80b-cc3f33fb6f14-kube-api-access-tl8rx" pod "969f6f63-470f-4b22-b80b-cc3f33fb6f14" (UID: "969f6f63-470f-4b22-b80b-cc3f33fb6f14"). InnerVolumeSpecName "kube-api-access-tl8rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:26.849230 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.849202    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-kube-api-access-g967m" pod "ff6ee21d-7956-4d2c-b00a-8e38afe9c585" (UID: "ff6ee21d-7956-4d2c-b00a-8e38afe9c585"). InnerVolumeSpecName "kube-api-access-g967m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:26.946257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.946224    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969f6f63-470f-4b22-b80b-cc3f33fb6f14-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:26.946257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.946253    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl8rx\" (UniqueName: \"kubernetes.io/projected/969f6f63-470f-4b22-b80b-cc3f33fb6f14-kube-api-access-tl8rx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:26.946257 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.946264    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g967m\" (UniqueName: \"kubernetes.io/projected/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-kube-api-access-g967m\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:26.946498 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.946273    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6ee21d-7956-4d2c-b00a-8e38afe9c585-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:26.954456 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.954427    4898 generic.go:356] "Generic (PLEG): container finished" podID="ff6ee21d-7956-4d2c-b00a-8e38afe9c585" containerID="963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859" exitCode=0
Nov 02 07:59:26.954589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.954476    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"
Nov 02 07:59:26.954589 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.954514    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l" event={"ID":"ff6ee21d-7956-4d2c-b00a-8e38afe9c585","Type":"ContainerDied","Data":"963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859"}
Nov 02 07:59:26.954725 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.954600    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l" event={"ID":"ff6ee21d-7956-4d2c-b00a-8e38afe9c585","Type":"ContainerDied","Data":"4b48cfae6d04d75dc57ff04abfbb148801e776c7610c2f2b54497ab0d0b0cdb4"}
Nov 02 07:59:26.954725 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.954633    4898 scope.go:122] "RemoveContainer" containerID="963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859"
Nov 02 07:59:26.958391 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.958363    4898 generic.go:356] "Generic (PLEG): container finished" podID="969f6f63-470f-4b22-b80b-cc3f33fb6f14" containerID="0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7" exitCode=0
Nov 02 07:59:26.959157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.959137    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"
Nov 02 07:59:26.962210 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.962189    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf" event={"ID":"969f6f63-470f-4b22-b80b-cc3f33fb6f14","Type":"ContainerDied","Data":"0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7"}
Nov 02 07:59:26.962325 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.962238    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf" event={"ID":"969f6f63-470f-4b22-b80b-cc3f33fb6f14","Type":"ContainerDied","Data":"8655ffd7b9f6253cdfcd26636fb978b7a65aa6d3b184df5b11d7d05dcba3bf64"}
Nov 02 07:59:26.970306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.970288    4898 scope.go:122] "RemoveContainer" containerID="963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859"
Nov 02 07:59:26.970595 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:59:26.970578    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859\": container with ID starting with 963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859 not found: ID does not exist" containerID="963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859"
Nov 02 07:59:26.970660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.970601    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859"} err="failed to get container status \"963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859\": rpc error: code = NotFound desc = could not find container \"963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859\": container with ID starting with 963acb24572bcd81ec494c4da4a652110bf7b5cf89be139cf3dc66f4b01c8859 not found: ID does not exist"
Nov 02 07:59:26.970660 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.970617    4898 scope.go:122] "RemoveContainer" containerID="0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7"
Nov 02 07:59:26.994336 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.994308    4898 scope.go:122] "RemoveContainer" containerID="0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7"
Nov 02 07:59:26.994706 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:59:26.994676    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7\": container with ID starting with 0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7 not found: ID does not exist" containerID="0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7"
Nov 02 07:59:26.994831 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:26.994780    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7"} err="failed to get container status \"0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7\": rpc error: code = NotFound desc = could not find container \"0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7\": container with ID starting with 0ce327a96d0e2f8a5ab03982c6560409299597143e419210863933cc96a8fac7 not found: ID does not exist"
Nov 02 07:59:27.033036 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.033013    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"]
Nov 02 07:59:27.071751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.071721    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ac716b-6b7ffb894f-dxs2l"]
Nov 02 07:59:27.124078 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.124047    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"]
Nov 02 07:59:27.127791 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.127768    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ac716b-7b9585cc95-zwtjf"]
Nov 02 07:59:27.466548 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.466514    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969f6f63-470f-4b22-b80b-cc3f33fb6f14" path="/var/lib/kubelet/pods/969f6f63-470f-4b22-b80b-cc3f33fb6f14/volumes"
Nov 02 07:59:27.467166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.467148    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff6ee21d-7956-4d2c-b00a-8e38afe9c585" path="/var/lib/kubelet/pods/ff6ee21d-7956-4d2c-b00a-8e38afe9c585/volumes"
Nov 02 07:59:27.893634 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.893573    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:59:27.959182 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.959155    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-socket\") pod \"534095d2-847d-4712-8f8d-acb19d774364\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") "
Nov 02 07:59:27.959364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.959200    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-envoy\") pod \"534095d2-847d-4712-8f8d-acb19d774364\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") "
Nov 02 07:59:27.959364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.959227    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-credential-socket\") pod \"534095d2-847d-4712-8f8d-acb19d774364\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") "
Nov 02 07:59:27.959364 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.959273    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-certs\") pod \"534095d2-847d-4712-8f8d-acb19d774364\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") "
Nov 02 07:59:27.959534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.959414    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-data\") pod \"534095d2-847d-4712-8f8d-acb19d774364\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") "
Nov 02 07:59:27.959534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.959460    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-socket" pod "534095d2-847d-4712-8f8d-acb19d774364" (UID: "534095d2-847d-4712-8f8d-acb19d774364"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:27.959534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.959496    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/534095d2-847d-4712-8f8d-acb19d774364-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534095d2-847d-4712-8f8d-acb19d774364-istiod-ca-cert\") pod \"534095d2-847d-4712-8f8d-acb19d774364\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") "
Nov 02 07:59:27.959696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.959620    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/534095d2-847d-4712-8f8d-acb19d774364-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534095d2-847d-4712-8f8d-acb19d774364-istio-podinfo\") pod \"534095d2-847d-4712-8f8d-acb19d774364\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") "
Nov 02 07:59:27.959696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.959655    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-istio-token\" (UniqueName: \"kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-istio-token\") pod \"534095d2-847d-4712-8f8d-acb19d774364\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") "
Nov 02 07:59:27.959696 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.959679    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-credential-socket" pod "534095d2-847d-4712-8f8d-acb19d774364" (UID: "534095d2-847d-4712-8f8d-acb19d774364"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:27.959854 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.959711    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-kube-api-access-hj4wz\" (UniqueName: \"kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-kube-api-access-hj4wz\") pod \"534095d2-847d-4712-8f8d-acb19d774364\" (UID: \"534095d2-847d-4712-8f8d-acb19d774364\") "
Nov 02 07:59:27.960376 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.960341    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:27.960376 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.960366    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:27.961722 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.960353    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534095d2-847d-4712-8f8d-acb19d774364-istiod-ca-cert" pod "534095d2-847d-4712-8f8d-acb19d774364" (UID: "534095d2-847d-4712-8f8d-acb19d774364"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:27.962051 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.962025    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-certs" pod "534095d2-847d-4712-8f8d-acb19d774364" (UID: "534095d2-847d-4712-8f8d-acb19d774364"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:27.962370 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.962348    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-data" pod "534095d2-847d-4712-8f8d-acb19d774364" (UID: "534095d2-847d-4712-8f8d-acb19d774364"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:27.962651 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.962549    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-envoy" pod "534095d2-847d-4712-8f8d-acb19d774364" (UID: "534095d2-847d-4712-8f8d-acb19d774364"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:27.962957 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.962884    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-kube-api-access-hj4wz" pod "534095d2-847d-4712-8f8d-acb19d774364" (UID: "534095d2-847d-4712-8f8d-acb19d774364"). InnerVolumeSpecName "kube-api-access-hj4wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:27.972534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.972511    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-istio-token" pod "534095d2-847d-4712-8f8d-acb19d774364" (UID: "534095d2-847d-4712-8f8d-acb19d774364"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:27.981254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.975223    4898 generic.go:356] "Generic (PLEG): container finished" podID="534095d2-847d-4712-8f8d-acb19d774364" containerID="03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f" exitCode=0
Nov 02 07:59:27.981254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.975435    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" event={"ID":"534095d2-847d-4712-8f8d-acb19d774364","Type":"ContainerDied","Data":"03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f"}
Nov 02 07:59:27.981254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.975483    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f" event={"ID":"534095d2-847d-4712-8f8d-acb19d774364","Type":"ContainerDied","Data":"49fa92c15594fc7fa722a3f15e209a9f087f23d355c295534941155d615c17be"}
Nov 02 07:59:27.981254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.975513    4898 scope.go:122] "RemoveContainer" containerID="03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f"
Nov 02 07:59:27.981254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.975624    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"
Nov 02 07:59:27.990185 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.990160    4898 scope.go:122] "RemoveContainer" containerID="03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f"
Nov 02 07:59:27.990834 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:59:27.990548    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f\": container with ID starting with 03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f not found: ID does not exist" containerID="03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f"
Nov 02 07:59:27.990834 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.990588    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f"} err="failed to get container status \"03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f\": rpc error: code = NotFound desc = could not find container \"03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f\": container with ID starting with 03e9658249febdef94945a4bf1ef58cd8b930e7e13f385a1229bfd2057412e0f not found: ID does not exist"
Nov 02 07:59:27.991299 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:27.991275    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/534095d2-847d-4712-8f8d-acb19d774364-istio-podinfo" pod "534095d2-847d-4712-8f8d-acb19d774364" (UID: "534095d2-847d-4712-8f8d-acb19d774364"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:59:28.068964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.068925    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534095d2-847d-4712-8f8d-acb19d774364-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:28.068964 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.068963    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:28.069197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.068977    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj4wz\" (UniqueName: \"kubernetes.io/projected/534095d2-847d-4712-8f8d-acb19d774364-kube-api-access-hj4wz\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:28.069197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.068991    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:28.069197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.069007    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:28.069197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.069019    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534095d2-847d-4712-8f8d-acb19d774364-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:28.069197 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.069034    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534095d2-847d-4712-8f8d-acb19d774364-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:28.324733 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.324699    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"]
Nov 02 07:59:28.331853 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.331823    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-337764-openshift-default-555d885b9-f5q9f"]
Nov 02 07:59:28.362385 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.362359    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"]
Nov 02 07:59:28.363013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.362996    4898 cpu_manager.go:414] "containerMap: removing container" podUID="969f6f63-470f-4b22-b80b-cc3f33fb6f14" containerName="payload-pre-processing"
Nov 02 07:59:28.363096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.363013    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969f6f63-470f-4b22-b80b-cc3f33fb6f14" containerName="payload-pre-processing"
Nov 02 07:59:28.363096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.363024    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ff6ee21d-7956-4d2c-b00a-8e38afe9c585" containerName="payload-processing"
Nov 02 07:59:28.363096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.363029    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6ee21d-7956-4d2c-b00a-8e38afe9c585" containerName="payload-processing"
Nov 02 07:59:28.363096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.363042    4898 cpu_manager.go:414] "containerMap: removing container" podUID="534095d2-847d-4712-8f8d-acb19d774364" containerName="istio-proxy"
Nov 02 07:59:28.363096 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.363047    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534095d2-847d-4712-8f8d-acb19d774364" containerName="istio-proxy"
Nov 02 07:59:28.363370 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.363155    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="969f6f63-470f-4b22-b80b-cc3f33fb6f14" containerName="payload-pre-processing"
Nov 02 07:59:28.363370 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.363164    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6ee21d-7956-4d2c-b00a-8e38afe9c585" containerName="payload-processing"
Nov 02 07:59:28.363370 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.363173    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="534095d2-847d-4712-8f8d-acb19d774364" containerName="istio-proxy"
Nov 02 07:59:28.368150 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.368121    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"
Nov 02 07:59:28.388307 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.388280    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-696c7a\""
Nov 02 07:59:28.388465 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.388449    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-696c7a-dockercfg-v4q25\""
Nov 02 07:59:28.393040 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.393015    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"]
Nov 02 07:59:28.483911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.477829    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dksp\" (UniqueName: \"kubernetes.io/projected/4e7cea32-8481-4054-89c1-6b72c41ea4f7-kube-api-access-9dksp\") pod \"payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f\" (UID: \"4e7cea32-8481-4054-89c1-6b72c41ea4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"
Nov 02 07:59:28.483911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.477912    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7cea32-8481-4054-89c1-6b72c41ea4f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f\" (UID: \"4e7cea32-8481-4054-89c1-6b72c41ea4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"
Nov 02 07:59:28.583033 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.582938    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dksp\" (UniqueName: \"kubernetes.io/projected/4e7cea32-8481-4054-89c1-6b72c41ea4f7-kube-api-access-9dksp\") pod \"payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f\" (UID: \"4e7cea32-8481-4054-89c1-6b72c41ea4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"
Nov 02 07:59:28.583627 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.583598    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7cea32-8481-4054-89c1-6b72c41ea4f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f\" (UID: \"4e7cea32-8481-4054-89c1-6b72c41ea4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"
Nov 02 07:59:28.584761 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.584630    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7cea32-8481-4054-89c1-6b72c41ea4f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f\" (UID: \"4e7cea32-8481-4054-89c1-6b72c41ea4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"
Nov 02 07:59:28.686989 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.686590    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dksp\" (UniqueName: \"kubernetes.io/projected/4e7cea32-8481-4054-89c1-6b72c41ea4f7-kube-api-access-9dksp\") pod \"payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f\" (UID: \"4e7cea32-8481-4054-89c1-6b72c41ea4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"
Nov 02 07:59:28.985157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:28.985119    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"
Nov 02 07:59:29.304156 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:59:29.304099    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7cea32_8481_4054_89c1_6b72c41ea4f7.slice/crio-336299f82b54b61f06e76bb95977a41bf4fc7b95ad67b9591156f04816c49c88 WatchSource:0}: Error finding container 336299f82b54b61f06e76bb95977a41bf4fc7b95ad67b9591156f04816c49c88: Status 404 returned error can't find the container with id 336299f82b54b61f06e76bb95977a41bf4fc7b95ad67b9591156f04816c49c88
Nov 02 07:59:29.307040 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:29.307017    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"]
Nov 02 07:59:29.475030 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:29.474998    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534095d2-847d-4712-8f8d-acb19d774364" path="/var/lib/kubelet/pods/534095d2-847d-4712-8f8d-acb19d774364/volumes"
Nov 02 07:59:29.999580 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:29.999545    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f" event={"ID":"4e7cea32-8481-4054-89c1-6b72c41ea4f7","Type":"ContainerStarted","Data":"a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce"}
Nov 02 07:59:29.999771 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:29.999597    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f" event={"ID":"4e7cea32-8481-4054-89c1-6b72c41ea4f7","Type":"ContainerStarted","Data":"336299f82b54b61f06e76bb95977a41bf4fc7b95ad67b9591156f04816c49c88"}
Nov 02 07:59:29.999771 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:29.999636    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"
Nov 02 07:59:30.051157 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:30.051070    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f" podStartSLOduration=2.051055644 podStartE2EDuration="2.051055644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:30.047104893 +0000 UTC m=+2120.737431856" watchObservedRunningTime="2026-11-02 07:59:30.051055644 +0000 UTC m=+2120.741382608"
Nov 02 07:59:33.932179 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:33.932140    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"]
Nov 02 07:59:33.932641 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:33.932346    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" podUID="7198de3d-596e-4e43-a71c-d1dcf8deea73" containerName="istio-proxy" containerID="cri-o://b32ef8138c6e5b6366eb55f8b6e228962a23d5b32c036ea0bf25a612a003f88c" gracePeriod=30
Nov 02 07:59:35.004421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.004380    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"]
Nov 02 07:59:35.009146 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.009118    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:35.027572 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.027542    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"]
Nov 02 07:59:35.031138 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.031110    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:59:35.031285 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.031215    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:59:35.034973 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.034432    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:59:35.034973 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.034590    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:59:35.099074 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.099041    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b73536d-6t8tt\" (UID: \"4c2f0387-92d1-464d-b3be-eb584e706018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:35.099256 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.099174    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wld67\" (UniqueName: \"kubernetes.io/projected/4c2f0387-92d1-464d-b3be-eb584e706018-kube-api-access-wld67\") pod \"maas-api-revoke-sub-0b73536d-6t8tt\" (UID: \"4c2f0387-92d1-464d-b3be-eb584e706018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:35.099335 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.099321    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b73536d-6t8tt\" (UID: \"4c2f0387-92d1-464d-b3be-eb584e706018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:35.201124 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.201085    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b73536d-6t8tt\" (UID: \"4c2f0387-92d1-464d-b3be-eb584e706018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:35.201330 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.201227    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wld67\" (UniqueName: \"kubernetes.io/projected/4c2f0387-92d1-464d-b3be-eb584e706018-kube-api-access-wld67\") pod \"maas-api-revoke-sub-0b73536d-6t8tt\" (UID: \"4c2f0387-92d1-464d-b3be-eb584e706018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:35.201422 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.201404    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b73536d-6t8tt\" (UID: \"4c2f0387-92d1-464d-b3be-eb584e706018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:35.202131 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.202105    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b73536d-6t8tt\" (UID: \"4c2f0387-92d1-464d-b3be-eb584e706018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:35.203809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.203776    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b73536d-6t8tt\" (UID: \"4c2f0387-92d1-464d-b3be-eb584e706018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:35.224670 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.224641    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wld67\" (UniqueName: \"kubernetes.io/projected/4c2f0387-92d1-464d-b3be-eb584e706018-kube-api-access-wld67\") pod \"maas-api-revoke-sub-0b73536d-6t8tt\" (UID: \"4c2f0387-92d1-464d-b3be-eb584e706018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:35.325103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.325025    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:35.634798 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:59:35.634766    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2f0387_92d1_464d_b3be_eb584e706018.slice/crio-bf8a81e95b6e63df2170f447188349f67b91dc350ba8d99d05fa16df32adc899 WatchSource:0}: Error finding container bf8a81e95b6e63df2170f447188349f67b91dc350ba8d99d05fa16df32adc899: Status 404 returned error can't find the container with id bf8a81e95b6e63df2170f447188349f67b91dc350ba8d99d05fa16df32adc899
Nov 02 07:59:35.637526 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:35.637500    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"]
Nov 02 07:59:36.087213 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.087184    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt" event={"ID":"4c2f0387-92d1-464d-b3be-eb584e706018","Type":"ContainerStarted","Data":"3b9f1be42dbdc54b713d5ee077690ebdbfe51b5b1b96ddb89887fa3b653a6ab6"}
Nov 02 07:59:36.087639 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.087250    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt" event={"ID":"4c2f0387-92d1-464d-b3be-eb584e706018","Type":"ContainerStarted","Data":"bf8a81e95b6e63df2170f447188349f67b91dc350ba8d99d05fa16df32adc899"}
Nov 02 07:59:36.141141 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.141091    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt" podStartSLOduration=2.14107893 podStartE2EDuration="2.14107893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:36.140215531 +0000 UTC m=+2126.830542517" watchObservedRunningTime="2026-11-02 07:59:36.14107893 +0000 UTC m=+2126.831405893"
Nov 02 07:59:36.277433 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.277392    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"]
Nov 02 07:59:36.283014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.282991    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:36.301065 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.301035    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"]
Nov 02 07:59:36.319306 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.319230    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhjw7\" (UniqueName: \"kubernetes.io/projected/ea00cdda-b2c6-4e94-bb14-12c924b61b79-kube-api-access-rhjw7\") pod \"maas-api-revoke-sub-7899d9f9-dw996\" (UID: \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:36.319427 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.319387    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7899d9f9-dw996\" (UID: \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:36.319637 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.319611    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7899d9f9-dw996\" (UID: \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:36.420978 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.420942    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhjw7\" (UniqueName: \"kubernetes.io/projected/ea00cdda-b2c6-4e94-bb14-12c924b61b79-kube-api-access-rhjw7\") pod \"maas-api-revoke-sub-7899d9f9-dw996\" (UID: \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:36.421128 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.421015    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7899d9f9-dw996\" (UID: \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:36.421176 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.421143    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7899d9f9-dw996\" (UID: \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:36.421686 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.421661    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7899d9f9-dw996\" (UID: \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:36.423520 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.423501    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7899d9f9-dw996\" (UID: \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:36.449844 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.449819    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhjw7\" (UniqueName: \"kubernetes.io/projected/ea00cdda-b2c6-4e94-bb14-12c924b61b79-kube-api-access-rhjw7\") pod \"maas-api-revoke-sub-7899d9f9-dw996\" (UID: \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:36.603764 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.603732    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:36.938655 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:59:36.938627    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea00cdda_b2c6_4e94_bb14_12c924b61b79.slice/crio-d997edecf345c5b7f95be28bd6e5586f4e44379c6551c413ef78475b6792cb30 WatchSource:0}: Error finding container d997edecf345c5b7f95be28bd6e5586f4e44379c6551c413ef78475b6792cb30: Status 404 returned error can't find the container with id d997edecf345c5b7f95be28bd6e5586f4e44379c6551c413ef78475b6792cb30
Nov 02 07:59:36.940076 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:36.940046    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"]
Nov 02 07:59:37.064883 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:37.064850    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"]
Nov 02 07:59:37.065133 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:37.065105    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" containerName="istio-proxy" containerID="cri-o://c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42" gracePeriod=30
Nov 02 07:59:37.098960 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:37.098871    4898 generic.go:356] "Generic (PLEG): container finished" podID="4c2f0387-92d1-464d-b3be-eb584e706018" containerID="3b9f1be42dbdc54b713d5ee077690ebdbfe51b5b1b96ddb89887fa3b653a6ab6" exitCode=0
Nov 02 07:59:37.099402 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:37.098933    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt" event={"ID":"4c2f0387-92d1-464d-b3be-eb584e706018","Type":"ContainerDied","Data":"3b9f1be42dbdc54b713d5ee077690ebdbfe51b5b1b96ddb89887fa3b653a6ab6"}
Nov 02 07:59:37.100728 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:37.100704    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996" event={"ID":"ea00cdda-b2c6-4e94-bb14-12c924b61b79","Type":"ContainerStarted","Data":"d997edecf345c5b7f95be28bd6e5586f4e44379c6551c413ef78475b6792cb30"}
Nov 02 07:59:38.110619 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:38.110585    4898 generic.go:356] "Generic (PLEG): container finished" podID="ea00cdda-b2c6-4e94-bb14-12c924b61b79" containerID="5d43cdc91bc558692e8ce405c9811bf874007404526fc2249d892639d4b5e48d" exitCode=0
Nov 02 07:59:38.111084 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:38.110670    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996" event={"ID":"ea00cdda-b2c6-4e94-bb14-12c924b61b79","Type":"ContainerDied","Data":"5d43cdc91bc558692e8ce405c9811bf874007404526fc2249d892639d4b5e48d"}
Nov 02 07:59:38.541115 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:38.541059    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:38.556217 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:38.556183    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-xks-ca\") pod \"4c2f0387-92d1-464d-b3be-eb584e706018\" (UID: \"4c2f0387-92d1-464d-b3be-eb584e706018\") "
Nov 02 07:59:38.556439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:38.556272    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-openshift-ca\") pod \"4c2f0387-92d1-464d-b3be-eb584e706018\" (UID: \"4c2f0387-92d1-464d-b3be-eb584e706018\") "
Nov 02 07:59:38.556439 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:38.556434    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2f0387-92d1-464d-b3be-eb584e706018-kube-api-access-wld67\" (UniqueName: \"kubernetes.io/projected/4c2f0387-92d1-464d-b3be-eb584e706018-kube-api-access-wld67\") pod \"4c2f0387-92d1-464d-b3be-eb584e706018\" (UID: \"4c2f0387-92d1-464d-b3be-eb584e706018\") "
Nov 02 07:59:38.556667 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:38.556634    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-openshift-ca" pod "4c2f0387-92d1-464d-b3be-eb584e706018" (UID: "4c2f0387-92d1-464d-b3be-eb584e706018"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:38.557002 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:38.556980    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:38.558668 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:38.558645    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-xks-ca" pod "4c2f0387-92d1-464d-b3be-eb584e706018" (UID: "4c2f0387-92d1-464d-b3be-eb584e706018"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:59:38.559537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:38.559517    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2f0387-92d1-464d-b3be-eb584e706018-kube-api-access-wld67" pod "4c2f0387-92d1-464d-b3be-eb584e706018" (UID: "4c2f0387-92d1-464d-b3be-eb584e706018"). InnerVolumeSpecName "kube-api-access-wld67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:38.658245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:38.658211    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2f0387-92d1-464d-b3be-eb584e706018-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:38.658245 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:38.658242    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wld67\" (UniqueName: \"kubernetes.io/projected/4c2f0387-92d1-464d-b3be-eb584e706018-kube-api-access-wld67\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:39.123795 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.123761    4898 generic.go:356] "Generic (PLEG): container finished" podID="7198de3d-596e-4e43-a71c-d1dcf8deea73" containerID="b32ef8138c6e5b6366eb55f8b6e228962a23d5b32c036ea0bf25a612a003f88c" exitCode=0
Nov 02 07:59:39.124312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.123925    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" event={"ID":"7198de3d-596e-4e43-a71c-d1dcf8deea73","Type":"ContainerDied","Data":"b32ef8138c6e5b6366eb55f8b6e228962a23d5b32c036ea0bf25a612a003f88c"}
Nov 02 07:59:39.126354 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.126326    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"
Nov 02 07:59:39.127005 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.126979    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt" event={"ID":"4c2f0387-92d1-464d-b3be-eb584e706018","Type":"ContainerDied","Data":"bf8a81e95b6e63df2170f447188349f67b91dc350ba8d99d05fa16df32adc899"}
Nov 02 07:59:39.127144 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.127132    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf8a81e95b6e63df2170f447188349f67b91dc350ba8d99d05fa16df32adc899"
Nov 02 07:59:39.200042 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.199945    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:59:39.272104 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.272068    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-kube-api-access-85nfd\" (UniqueName: \"kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-kube-api-access-85nfd\") pod \"7198de3d-596e-4e43-a71c-d1dcf8deea73\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") "
Nov 02 07:59:39.272287 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.272117    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-token\" (UniqueName: \"kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-token\") pod \"7198de3d-596e-4e43-a71c-d1dcf8deea73\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") "
Nov 02 07:59:39.272287 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.272159    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-data\") pod \"7198de3d-596e-4e43-a71c-d1dcf8deea73\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") "
Nov 02 07:59:39.272287 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.272231    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-credential-socket\") pod \"7198de3d-596e-4e43-a71c-d1dcf8deea73\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") "
Nov 02 07:59:39.272481 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.272319    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-envoy\") pod \"7198de3d-596e-4e43-a71c-d1dcf8deea73\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") "
Nov 02 07:59:39.272481 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.272363    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-podinfo\") pod \"7198de3d-596e-4e43-a71c-d1dcf8deea73\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") "
Nov 02 07:59:39.272591 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.272490    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-socket\") pod \"7198de3d-596e-4e43-a71c-d1dcf8deea73\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") "
Nov 02 07:59:39.272759 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.272737    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-socket" pod "7198de3d-596e-4e43-a71c-d1dcf8deea73" (UID: "7198de3d-596e-4e43-a71c-d1dcf8deea73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:39.273172 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.272971    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-data" pod "7198de3d-596e-4e43-a71c-d1dcf8deea73" (UID: "7198de3d-596e-4e43-a71c-d1dcf8deea73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:39.276037 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.275988    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-credential-socket" pod "7198de3d-596e-4e43-a71c-d1dcf8deea73" (UID: "7198de3d-596e-4e43-a71c-d1dcf8deea73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:39.276189 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.276036    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-certs\") pod \"7198de3d-596e-4e43-a71c-d1dcf8deea73\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") "
Nov 02 07:59:39.276189 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.276123    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7198de3d-596e-4e43-a71c-d1dcf8deea73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7198de3d-596e-4e43-a71c-d1dcf8deea73-istiod-ca-cert\") pod \"7198de3d-596e-4e43-a71c-d1dcf8deea73\" (UID: \"7198de3d-596e-4e43-a71c-d1dcf8deea73\") "
Nov 02 07:59:39.277073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.276600    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-podinfo" pod "7198de3d-596e-4e43-a71c-d1dcf8deea73" (UID: "7198de3d-596e-4e43-a71c-d1dcf8deea73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:59:39.277073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.276811    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-certs" pod "7198de3d-596e-4e43-a71c-d1dcf8deea73" (UID: "7198de3d-596e-4e43-a71c-d1dcf8deea73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:39.277073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.276857    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:39.277073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.276875    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:39.277073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.276930    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:39.277073 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.276946    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:39.277425 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.277254    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7198de3d-596e-4e43-a71c-d1dcf8deea73-istiod-ca-cert" pod "7198de3d-596e-4e43-a71c-d1dcf8deea73" (UID: "7198de3d-596e-4e43-a71c-d1dcf8deea73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:39.280107 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.280073    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-token" pod "7198de3d-596e-4e43-a71c-d1dcf8deea73" (UID: "7198de3d-596e-4e43-a71c-d1dcf8deea73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:39.281207 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.281174    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-kube-api-access-85nfd" pod "7198de3d-596e-4e43-a71c-d1dcf8deea73" (UID: "7198de3d-596e-4e43-a71c-d1dcf8deea73"). InnerVolumeSpecName "kube-api-access-85nfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:39.282082 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.282063    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-envoy" pod "7198de3d-596e-4e43-a71c-d1dcf8deea73" (UID: "7198de3d-596e-4e43-a71c-d1dcf8deea73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:39.300389 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.300293    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:39.387862 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.378144    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-xks-ca\") pod \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\" (UID: \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\") "
Nov 02 07:59:39.389562 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.389359    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-openshift-ca\") pod \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\" (UID: \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\") "
Nov 02 07:59:39.392032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.390013    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea00cdda-b2c6-4e94-bb14-12c924b61b79-kube-api-access-rhjw7\" (UniqueName: \"kubernetes.io/projected/ea00cdda-b2c6-4e94-bb14-12c924b61b79-kube-api-access-rhjw7\") pod \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\" (UID: \"ea00cdda-b2c6-4e94-bb14-12c924b61b79\") "
Nov 02 07:59:39.392032 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.390157    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-openshift-ca" pod "ea00cdda-b2c6-4e94-bb14-12c924b61b79" (UID: "ea00cdda-b2c6-4e94-bb14-12c924b61b79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:39.394093 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.394066    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:39.394183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.394097    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:39.394183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.394114    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7198de3d-596e-4e43-a71c-d1dcf8deea73-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:39.394183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.394130    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7198de3d-596e-4e43-a71c-d1dcf8deea73-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:39.394183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.394144    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85nfd\" (UniqueName: \"kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-kube-api-access-85nfd\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:39.394183 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.394157    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7198de3d-596e-4e43-a71c-d1dcf8deea73-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:39.394415 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.394319    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea00cdda-b2c6-4e94-bb14-12c924b61b79-kube-api-access-rhjw7" pod "ea00cdda-b2c6-4e94-bb14-12c924b61b79" (UID: "ea00cdda-b2c6-4e94-bb14-12c924b61b79"). InnerVolumeSpecName "kube-api-access-rhjw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:39.395982 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.395957    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-xks-ca" pod "ea00cdda-b2c6-4e94-bb14-12c924b61b79" (UID: "ea00cdda-b2c6-4e94-bb14-12c924b61b79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:59:39.495537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.495499    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhjw7\" (UniqueName: \"kubernetes.io/projected/ea00cdda-b2c6-4e94-bb14-12c924b61b79-kube-api-access-rhjw7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:39.495537 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:39.495527    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea00cdda-b2c6-4e94-bb14-12c924b61b79-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:40.134736 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:40.134707    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"
Nov 02 07:59:40.135170 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:40.134757    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996" event={"ID":"ea00cdda-b2c6-4e94-bb14-12c924b61b79","Type":"ContainerDied","Data":"d997edecf345c5b7f95be28bd6e5586f4e44379c6551c413ef78475b6792cb30"}
Nov 02 07:59:40.135170 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:40.134804    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d997edecf345c5b7f95be28bd6e5586f4e44379c6551c413ef78475b6792cb30"
Nov 02 07:59:40.136438 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:40.136420    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"
Nov 02 07:59:40.136594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:40.136463    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf" event={"ID":"7198de3d-596e-4e43-a71c-d1dcf8deea73","Type":"ContainerDied","Data":"202d5a84ab844a93f62fec3200457dd717f37f3157fd80749815c71b27c1fac1"}
Nov 02 07:59:40.136594 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:40.136515    4898 scope.go:122] "RemoveContainer" containerID="b32ef8138c6e5b6366eb55f8b6e228962a23d5b32c036ea0bf25a612a003f88c"
Nov 02 07:59:40.222260 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:40.222227    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"]
Nov 02 07:59:40.234047 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:40.234015    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-tzjqf"]
Nov 02 07:59:41.032751 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:41.032719    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"
Nov 02 07:59:41.466560 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:41.466524    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7198de3d-596e-4e43-a71c-d1dcf8deea73" path="/var/lib/kubelet/pods/7198de3d-596e-4e43-a71c-d1dcf8deea73/volumes"
Nov 02 07:59:42.363574 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.363548    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:59:42.443612 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.443583    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-kube-api-access-sn7h6\" (UniqueName: \"kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-kube-api-access-sn7h6\") pod \"54d8dd92-580e-4e7c-9114-1215f2100400\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") "
Nov 02 07:59:42.443799 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.443658    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-data\") pod \"54d8dd92-580e-4e7c-9114-1215f2100400\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") "
Nov 02 07:59:42.443799 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.443697    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-certs\") pod \"54d8dd92-580e-4e7c-9114-1215f2100400\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") "
Nov 02 07:59:42.443969 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.443816    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54d8dd92-580e-4e7c-9114-1215f2100400-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d8dd92-580e-4e7c-9114-1215f2100400-istio-podinfo\") pod \"54d8dd92-580e-4e7c-9114-1215f2100400\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") "
Nov 02 07:59:42.443969 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.443846    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-credential-socket\") pod \"54d8dd92-580e-4e7c-9114-1215f2100400\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") "
Nov 02 07:59:42.444101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.443970    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54d8dd92-580e-4e7c-9114-1215f2100400-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d8dd92-580e-4e7c-9114-1215f2100400-istiod-ca-cert\") pod \"54d8dd92-580e-4e7c-9114-1215f2100400\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") "
Nov 02 07:59:42.444101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.444001    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-data" pod "54d8dd92-580e-4e7c-9114-1215f2100400" (UID: "54d8dd92-580e-4e7c-9114-1215f2100400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:42.444101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.444018    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-certs" pod "54d8dd92-580e-4e7c-9114-1215f2100400" (UID: "54d8dd92-580e-4e7c-9114-1215f2100400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:42.444101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.444047    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-envoy\") pod \"54d8dd92-580e-4e7c-9114-1215f2100400\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") "
Nov 02 07:59:42.444101 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.444093    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-istio-token\" (UniqueName: \"kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-istio-token\") pod \"54d8dd92-580e-4e7c-9114-1215f2100400\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") "
Nov 02 07:59:42.444377 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.444170    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-socket\") pod \"54d8dd92-580e-4e7c-9114-1215f2100400\" (UID: \"54d8dd92-580e-4e7c-9114-1215f2100400\") "
Nov 02 07:59:42.444377 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.444227    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-credential-socket" pod "54d8dd92-580e-4e7c-9114-1215f2100400" (UID: "54d8dd92-580e-4e7c-9114-1215f2100400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:42.444478 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.444381    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d8dd92-580e-4e7c-9114-1215f2100400-istiod-ca-cert" pod "54d8dd92-580e-4e7c-9114-1215f2100400" (UID: "54d8dd92-580e-4e7c-9114-1215f2100400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:42.445376 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.445291    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-socket" pod "54d8dd92-580e-4e7c-9114-1215f2100400" (UID: "54d8dd92-580e-4e7c-9114-1215f2100400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:42.446329 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.445766    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:42.446467 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.446343    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:42.446551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.446498    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:42.446551 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.446515    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:42.446797 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.446531    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d8dd92-580e-4e7c-9114-1215f2100400-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:42.446879 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.446836    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54d8dd92-580e-4e7c-9114-1215f2100400-istio-podinfo" pod "54d8dd92-580e-4e7c-9114-1215f2100400" (UID: "54d8dd92-580e-4e7c-9114-1215f2100400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:59:42.446998 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.446913    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-envoy" pod "54d8dd92-580e-4e7c-9114-1215f2100400" (UID: "54d8dd92-580e-4e7c-9114-1215f2100400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:42.448603 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.448577    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-kube-api-access-sn7h6" pod "54d8dd92-580e-4e7c-9114-1215f2100400" (UID: "54d8dd92-580e-4e7c-9114-1215f2100400"). InnerVolumeSpecName "kube-api-access-sn7h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:42.449103 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.448686    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-istio-token" pod "54d8dd92-580e-4e7c-9114-1215f2100400" (UID: "54d8dd92-580e-4e7c-9114-1215f2100400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:42.547509 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.547475    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d8dd92-580e-4e7c-9114-1215f2100400-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:42.547509 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.547507    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:42.547509 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.547518    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn7h6\" (UniqueName: \"kubernetes.io/projected/54d8dd92-580e-4e7c-9114-1215f2100400-kube-api-access-sn7h6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:42.548014 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:42.547528    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d8dd92-580e-4e7c-9114-1215f2100400-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:43.166550 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.166511    4898 generic.go:356] "Generic (PLEG): container finished" podID="54d8dd92-580e-4e7c-9114-1215f2100400" containerID="c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42" exitCode=0
Nov 02 07:59:43.166723 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.166558    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"
Nov 02 07:59:43.166723 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.166583    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" event={"ID":"54d8dd92-580e-4e7c-9114-1215f2100400","Type":"ContainerDied","Data":"c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42"}
Nov 02 07:59:43.166723 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.166648    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq" event={"ID":"54d8dd92-580e-4e7c-9114-1215f2100400","Type":"ContainerDied","Data":"c05b7315e4905b3387e42e40cab6d896713f792f91b5d0701f4e0c7f1fb7663c"}
Nov 02 07:59:43.166723 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.166676    4898 scope.go:122] "RemoveContainer" containerID="c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42"
Nov 02 07:59:43.178329 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.178314    4898 scope.go:122] "RemoveContainer" containerID="a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327"
Nov 02 07:59:43.188301 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.188134    4898 scope.go:122] "RemoveContainer" containerID="c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42"
Nov 02 07:59:43.188441 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:59:43.188424    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42\": container with ID starting with c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42 not found: ID does not exist" containerID="c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42"
Nov 02 07:59:43.188499 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.188448    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42"} err="failed to get container status \"c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42\": rpc error: code = NotFound desc = could not find container \"c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42\": container with ID starting with c771f16b15500690b5268f9c32230811c978c865625ab3998cdb4c79a5f6ab42 not found: ID does not exist"
Nov 02 07:59:43.188499 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.188465    4898 scope.go:122] "RemoveContainer" containerID="a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327"
Nov 02 07:59:43.188713 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:59:43.188694    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327\": container with ID starting with a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327 not found: ID does not exist" containerID="a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327"
Nov 02 07:59:43.188805 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.188723    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327"} err="failed to get container status \"a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327\": rpc error: code = NotFound desc = could not find container \"a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327\": container with ID starting with a500709107005256be4b9b52ee130aad3eba0b482a6c22a1717dd9200d1f7327 not found: ID does not exist"
Nov 02 07:59:43.245323 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.245298    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"]
Nov 02 07:59:43.253235 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.253212    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-ac716b-openshift-default-5c495d47c-bggbq"]
Nov 02 07:59:43.466666 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:43.466588    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" path="/var/lib/kubelet/pods/54d8dd92-580e-4e7c-9114-1215f2100400/volumes"
Nov 02 07:59:44.989755 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:59:44.989714    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 07:59:45.382642 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.382560    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"]
Nov 02 07:59:45.382919 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.382853    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" podUID="dbe2608e-0a74-4371-9bc7-8425fb955660" containerName="istio-proxy" containerID="cri-o://3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738" gracePeriod=30
Nov 02 07:59:45.659614 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.659583    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"]
Nov 02 07:59:45.660234 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660218    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7198de3d-596e-4e43-a71c-d1dcf8deea73" containerName="istio-proxy"
Nov 02 07:59:45.660286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660235    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7198de3d-596e-4e43-a71c-d1dcf8deea73" containerName="istio-proxy"
Nov 02 07:59:45.660286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660248    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ea00cdda-b2c6-4e94-bb14-12c924b61b79" containerName="revoke-keys"
Nov 02 07:59:45.660286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660253    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea00cdda-b2c6-4e94-bb14-12c924b61b79" containerName="revoke-keys"
Nov 02 07:59:45.660286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660271    4898 cpu_manager.go:414] "containerMap: removing container" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" containerName="istio-proxy"
Nov 02 07:59:45.660286 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660276    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" containerName="istio-proxy"
Nov 02 07:59:45.660450 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660304    4898 cpu_manager.go:414] "containerMap: removing container" podUID="4c2f0387-92d1-464d-b3be-eb584e706018" containerName="revoke-keys"
Nov 02 07:59:45.660450 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660308    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2f0387-92d1-464d-b3be-eb584e706018" containerName="revoke-keys"
Nov 02 07:59:45.660450 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660406    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2f0387-92d1-464d-b3be-eb584e706018" containerName="revoke-keys"
Nov 02 07:59:45.660450 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660423    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7198de3d-596e-4e43-a71c-d1dcf8deea73" containerName="istio-proxy"
Nov 02 07:59:45.660450 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660433    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea00cdda-b2c6-4e94-bb14-12c924b61b79" containerName="revoke-keys"
Nov 02 07:59:45.660450 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660443    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" containerName="istio-proxy"
Nov 02 07:59:45.660450 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.660451    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" containerName="istio-proxy"
Nov 02 07:59:45.665645 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.665624    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:45.696966 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.696939    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 07:59:45.697277 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.697258    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 07:59:45.697501 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.697485    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 07:59:45.697738 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.697658    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"]
Nov 02 07:59:45.697857 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.697704    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 07:59:45.805610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.805314    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssk5k\" (UniqueName: \"kubernetes.io/projected/d8147482-0f1f-4dbe-bf62-5886c237bfe4-kube-api-access-ssk5k\") pod \"maas-api-revoke-sub-44e0bb5d-fndhj\" (UID: \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:45.805610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.805482    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44e0bb5d-fndhj\" (UID: \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:45.805610 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.805592    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44e0bb5d-fndhj\" (UID: \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:45.907081 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.907042    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44e0bb5d-fndhj\" (UID: \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:45.907253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.907140    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44e0bb5d-fndhj\" (UID: \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:45.907253 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.907228    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssk5k\" (UniqueName: \"kubernetes.io/projected/d8147482-0f1f-4dbe-bf62-5886c237bfe4-kube-api-access-ssk5k\") pod \"maas-api-revoke-sub-44e0bb5d-fndhj\" (UID: \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:45.907809 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.907780    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44e0bb5d-fndhj\" (UID: \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:45.909484 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.909462    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44e0bb5d-fndhj\" (UID: \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:45.958590 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.958523    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssk5k\" (UniqueName: \"kubernetes.io/projected/d8147482-0f1f-4dbe-bf62-5886c237bfe4-kube-api-access-ssk5k\") pod \"maas-api-revoke-sub-44e0bb5d-fndhj\" (UID: \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:45.980374 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:45.980343    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:46.320984 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:59:46.320951    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8147482_0f1f_4dbe_bf62_5886c237bfe4.slice/crio-8680bd0e39d85aeb2c47f390e1b31b6defc58292047f0dae43e28f8637065592 WatchSource:0}: Error finding container 8680bd0e39d85aeb2c47f390e1b31b6defc58292047f0dae43e28f8637065592: Status 404 returned error can't find the container with id 8680bd0e39d85aeb2c47f390e1b31b6defc58292047f0dae43e28f8637065592
Nov 02 07:59:46.325448 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:46.325388    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"]
Nov 02 07:59:46.829968 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:46.829882    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"]
Nov 02 07:59:46.830524 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:46.830511    4898 cpu_manager.go:414] "containerMap: removing container" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" containerName="istio-proxy"
Nov 02 07:59:46.830569 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:46.830526    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d8dd92-580e-4e7c-9114-1215f2100400" containerName="istio-proxy"
Nov 02 07:59:46.834254 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:46.834237    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:46.874769 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:46.874740    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"]
Nov 02 07:59:46.921882 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:46.921842    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a310ec-nzvst\" (UID: \"69b71e78-38e8-491a-8806-88d576ab0696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:46.922040 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:46.921915    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a310ec-nzvst\" (UID: \"69b71e78-38e8-491a-8806-88d576ab0696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:46.922105 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:46.922048    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rszg6\" (UniqueName: \"kubernetes.io/projected/69b71e78-38e8-491a-8806-88d576ab0696-kube-api-access-rszg6\") pod \"maas-api-revoke-sub-a6a310ec-nzvst\" (UID: \"69b71e78-38e8-491a-8806-88d576ab0696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:47.023677 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:47.023641    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a310ec-nzvst\" (UID: \"69b71e78-38e8-491a-8806-88d576ab0696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:47.023850 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:47.023742    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rszg6\" (UniqueName: \"kubernetes.io/projected/69b71e78-38e8-491a-8806-88d576ab0696-kube-api-access-rszg6\") pod \"maas-api-revoke-sub-a6a310ec-nzvst\" (UID: \"69b71e78-38e8-491a-8806-88d576ab0696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:47.024053 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:47.024031    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a310ec-nzvst\" (UID: \"69b71e78-38e8-491a-8806-88d576ab0696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:47.024587 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:47.024563    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a310ec-nzvst\" (UID: \"69b71e78-38e8-491a-8806-88d576ab0696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:47.026239 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:47.026212    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a310ec-nzvst\" (UID: \"69b71e78-38e8-491a-8806-88d576ab0696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:47.052027 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:47.052000    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rszg6\" (UniqueName: \"kubernetes.io/projected/69b71e78-38e8-491a-8806-88d576ab0696-kube-api-access-rszg6\") pod \"maas-api-revoke-sub-a6a310ec-nzvst\" (UID: \"69b71e78-38e8-491a-8806-88d576ab0696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:47.149938 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:47.149843    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:47.212883 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:47.212432    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj" event={"ID":"d8147482-0f1f-4dbe-bf62-5886c237bfe4","Type":"ContainerStarted","Data":"13312acd61b87e2b179500a2042e41779470285ecd58481cdd7d033bcb9248c2"}
Nov 02 07:59:47.212883 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:47.212490    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj" event={"ID":"d8147482-0f1f-4dbe-bf62-5886c237bfe4","Type":"ContainerStarted","Data":"8680bd0e39d85aeb2c47f390e1b31b6defc58292047f0dae43e28f8637065592"}
Nov 02 07:59:47.272917 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:47.272592    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj" podStartSLOduration=2.272572381 podStartE2EDuration="2.272572381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:47.271272756 +0000 UTC m=+2137.961599724" watchObservedRunningTime="2026-11-02 07:59:47.272572381 +0000 UTC m=+2137.962899353"
Nov 02 07:59:47.477243 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:47.477215    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"]
Nov 02 07:59:47.477830 ip-10-0-134-79 kubenswrapper[4898]: W1102 07:59:47.477803    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b71e78_38e8_491a_8806_88d576ab0696.slice/crio-09beb8813000b7ae3422057c771c3143dc453ea5316ce1f3f8284a1b0f9a4cf9 WatchSource:0}: Error finding container 09beb8813000b7ae3422057c771c3143dc453ea5316ce1f3f8284a1b0f9a4cf9: Status 404 returned error can't find the container with id 09beb8813000b7ae3422057c771c3143dc453ea5316ce1f3f8284a1b0f9a4cf9
Nov 02 07:59:48.221615 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:48.221572    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst" event={"ID":"69b71e78-38e8-491a-8806-88d576ab0696","Type":"ContainerStarted","Data":"38c19ac316a0d5fb539fb92f0a18bfafe59f51758cf00e3fa54e7c14fcbf21b7"}
Nov 02 07:59:48.221792 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:48.221633    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst" event={"ID":"69b71e78-38e8-491a-8806-88d576ab0696","Type":"ContainerStarted","Data":"09beb8813000b7ae3422057c771c3143dc453ea5316ce1f3f8284a1b0f9a4cf9"}
Nov 02 07:59:48.223421 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:48.223393    4898 generic.go:356] "Generic (PLEG): container finished" podID="d8147482-0f1f-4dbe-bf62-5886c237bfe4" containerID="13312acd61b87e2b179500a2042e41779470285ecd58481cdd7d033bcb9248c2" exitCode=0
Nov 02 07:59:48.223557 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:48.223445    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj" event={"ID":"d8147482-0f1f-4dbe-bf62-5886c237bfe4","Type":"ContainerDied","Data":"13312acd61b87e2b179500a2042e41779470285ecd58481cdd7d033bcb9248c2"}
Nov 02 07:59:48.271481 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:48.271429    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst" podStartSLOduration=2.271417324 podStartE2EDuration="2.271417324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:59:48.270019496 +0000 UTC m=+2138.960346461" watchObservedRunningTime="2026-11-02 07:59:48.271417324 +0000 UTC m=+2138.961744287"
Nov 02 07:59:49.232902 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:49.232855    4898 generic.go:356] "Generic (PLEG): container finished" podID="69b71e78-38e8-491a-8806-88d576ab0696" containerID="38c19ac316a0d5fb539fb92f0a18bfafe59f51758cf00e3fa54e7c14fcbf21b7" exitCode=0
Nov 02 07:59:49.233340 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:49.232953    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst" event={"ID":"69b71e78-38e8-491a-8806-88d576ab0696","Type":"ContainerDied","Data":"38c19ac316a0d5fb539fb92f0a18bfafe59f51758cf00e3fa54e7c14fcbf21b7"}
Nov 02 07:59:49.400336 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:49.400313    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:49.565502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:49.565415    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-openshift-ca\") pod \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\" (UID: \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\") "
Nov 02 07:59:49.565502 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:49.565487    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8147482-0f1f-4dbe-bf62-5886c237bfe4-kube-api-access-ssk5k\" (UniqueName: \"kubernetes.io/projected/d8147482-0f1f-4dbe-bf62-5886c237bfe4-kube-api-access-ssk5k\") pod \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\" (UID: \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\") "
Nov 02 07:59:49.565731 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:49.565549    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-xks-ca\") pod \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\" (UID: \"d8147482-0f1f-4dbe-bf62-5886c237bfe4\") "
Nov 02 07:59:49.565841 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:49.565817    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-openshift-ca" pod "d8147482-0f1f-4dbe-bf62-5886c237bfe4" (UID: "d8147482-0f1f-4dbe-bf62-5886c237bfe4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:49.566275 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:49.566249    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:49.567869 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:49.567832    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-xks-ca" pod "d8147482-0f1f-4dbe-bf62-5886c237bfe4" (UID: "d8147482-0f1f-4dbe-bf62-5886c237bfe4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:59:49.568229 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:49.568211    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8147482-0f1f-4dbe-bf62-5886c237bfe4-kube-api-access-ssk5k" pod "d8147482-0f1f-4dbe-bf62-5886c237bfe4" (UID: "d8147482-0f1f-4dbe-bf62-5886c237bfe4"). InnerVolumeSpecName "kube-api-access-ssk5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:49.667779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:49.667741    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8147482-0f1f-4dbe-bf62-5886c237bfe4-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:49.667779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:49.667776    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssk5k\" (UniqueName: \"kubernetes.io/projected/d8147482-0f1f-4dbe-bf62-5886c237bfe4-kube-api-access-ssk5k\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:50.242296 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.242266    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"
Nov 02 07:59:50.242779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.242312    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj" event={"ID":"d8147482-0f1f-4dbe-bf62-5886c237bfe4","Type":"ContainerDied","Data":"8680bd0e39d85aeb2c47f390e1b31b6defc58292047f0dae43e28f8637065592"}
Nov 02 07:59:50.242779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.242374    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8680bd0e39d85aeb2c47f390e1b31b6defc58292047f0dae43e28f8637065592"
Nov 02 07:59:50.512932 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.512109    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:50.682911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.682414    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b71e78-38e8-491a-8806-88d576ab0696-kube-api-access-rszg6\" (UniqueName: \"kubernetes.io/projected/69b71e78-38e8-491a-8806-88d576ab0696-kube-api-access-rszg6\") pod \"69b71e78-38e8-491a-8806-88d576ab0696\" (UID: \"69b71e78-38e8-491a-8806-88d576ab0696\") "
Nov 02 07:59:50.682911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.682721    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-xks-ca\") pod \"69b71e78-38e8-491a-8806-88d576ab0696\" (UID: \"69b71e78-38e8-491a-8806-88d576ab0696\") "
Nov 02 07:59:50.682911 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.682770    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-openshift-ca\") pod \"69b71e78-38e8-491a-8806-88d576ab0696\" (UID: \"69b71e78-38e8-491a-8806-88d576ab0696\") "
Nov 02 07:59:50.684939 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.683690    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-openshift-ca" pod "69b71e78-38e8-491a-8806-88d576ab0696" (UID: "69b71e78-38e8-491a-8806-88d576ab0696"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:50.691007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.689019    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-xks-ca" pod "69b71e78-38e8-491a-8806-88d576ab0696" (UID: "69b71e78-38e8-491a-8806-88d576ab0696"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 07:59:50.691007 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.689469    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b71e78-38e8-491a-8806-88d576ab0696-kube-api-access-rszg6" pod "69b71e78-38e8-491a-8806-88d576ab0696" (UID: "69b71e78-38e8-491a-8806-88d576ab0696"). InnerVolumeSpecName "kube-api-access-rszg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:50.729779 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.729751    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:59:50.784858 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.784767    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:50.784858 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.784805    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b71e78-38e8-491a-8806-88d576ab0696-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:50.784858 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.784821    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rszg6\" (UniqueName: \"kubernetes.io/projected/69b71e78-38e8-491a-8806-88d576ab0696-kube-api-access-rszg6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:50.885756 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.885718    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-kube-api-access-5xzw7\" (UniqueName: \"kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-kube-api-access-5xzw7\") pod \"dbe2608e-0a74-4371-9bc7-8425fb955660\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") "
Nov 02 07:59:50.886013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.885771    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-token\" (UniqueName: \"kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-token\") pod \"dbe2608e-0a74-4371-9bc7-8425fb955660\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") "
Nov 02 07:59:50.886013 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.885985    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-data\") pod \"dbe2608e-0a74-4371-9bc7-8425fb955660\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") "
Nov 02 07:59:50.886140 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.886020    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-envoy\") pod \"dbe2608e-0a74-4371-9bc7-8425fb955660\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") "
Nov 02 07:59:50.886140 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.886035    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-certs\") pod \"dbe2608e-0a74-4371-9bc7-8425fb955660\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") "
Nov 02 07:59:50.886140 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.886055    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-podinfo\") pod \"dbe2608e-0a74-4371-9bc7-8425fb955660\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") "
Nov 02 07:59:50.886140 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.886096    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-socket\") pod \"dbe2608e-0a74-4371-9bc7-8425fb955660\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") "
Nov 02 07:59:50.886140 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.886120    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-credential-socket\") pod \"dbe2608e-0a74-4371-9bc7-8425fb955660\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") "
Nov 02 07:59:50.886406 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.886146    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe2608e-0a74-4371-9bc7-8425fb955660-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbe2608e-0a74-4371-9bc7-8425fb955660-istiod-ca-cert\") pod \"dbe2608e-0a74-4371-9bc7-8425fb955660\" (UID: \"dbe2608e-0a74-4371-9bc7-8425fb955660\") "
Nov 02 07:59:50.886534 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.886503    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-certs" pod "dbe2608e-0a74-4371-9bc7-8425fb955660" (UID: "dbe2608e-0a74-4371-9bc7-8425fb955660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:50.886794 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.886766    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:50.888702 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.888677    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-kube-api-access-5xzw7" pod "dbe2608e-0a74-4371-9bc7-8425fb955660" (UID: "dbe2608e-0a74-4371-9bc7-8425fb955660"). InnerVolumeSpecName "kube-api-access-5xzw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:50.889393 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.889344    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-data" pod "dbe2608e-0a74-4371-9bc7-8425fb955660" (UID: "dbe2608e-0a74-4371-9bc7-8425fb955660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:50.889629 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.889590    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-socket" pod "dbe2608e-0a74-4371-9bc7-8425fb955660" (UID: "dbe2608e-0a74-4371-9bc7-8425fb955660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:50.889710 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.889683    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-token" pod "dbe2608e-0a74-4371-9bc7-8425fb955660" (UID: "dbe2608e-0a74-4371-9bc7-8425fb955660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 07:59:50.889798 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.889775    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-podinfo" pod "dbe2608e-0a74-4371-9bc7-8425fb955660" (UID: "dbe2608e-0a74-4371-9bc7-8425fb955660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 07:59:50.889856 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.889782    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-credential-socket" pod "dbe2608e-0a74-4371-9bc7-8425fb955660" (UID: "dbe2608e-0a74-4371-9bc7-8425fb955660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:50.891312 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.891287    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe2608e-0a74-4371-9bc7-8425fb955660-istiod-ca-cert" pod "dbe2608e-0a74-4371-9bc7-8425fb955660" (UID: "dbe2608e-0a74-4371-9bc7-8425fb955660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 07:59:50.891690 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.891664    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-envoy" pod "dbe2608e-0a74-4371-9bc7-8425fb955660" (UID: "dbe2608e-0a74-4371-9bc7-8425fb955660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 07:59:50.987788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.987745    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:50.987788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.987778    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:50.987788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.987787    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:50.987788 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.987796    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:50.988166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.987805    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbe2608e-0a74-4371-9bc7-8425fb955660-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:50.988166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.987815    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbe2608e-0a74-4371-9bc7-8425fb955660-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:50.988166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.987825    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xzw7\" (UniqueName: \"kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-kube-api-access-5xzw7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:50.988166 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:50.987833    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbe2608e-0a74-4371-9bc7-8425fb955660-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 07:59:51.253298 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.253260    4898 generic.go:356] "Generic (PLEG): container finished" podID="dbe2608e-0a74-4371-9bc7-8425fb955660" containerID="3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738" exitCode=0
Nov 02 07:59:51.253726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.253309    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"
Nov 02 07:59:51.253726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.253334    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" event={"ID":"dbe2608e-0a74-4371-9bc7-8425fb955660","Type":"ContainerDied","Data":"3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738"}
Nov 02 07:59:51.253726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.253419    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm" event={"ID":"dbe2608e-0a74-4371-9bc7-8425fb955660","Type":"ContainerDied","Data":"afaa76bbfc1ea7e5d21ec7b22c8add241cb5c04eea186eb6c6c30ab96be87d29"}
Nov 02 07:59:51.253726 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.253451    4898 scope.go:122] "RemoveContainer" containerID="3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738"
Nov 02 07:59:51.255573 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.255547    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst" event={"ID":"69b71e78-38e8-491a-8806-88d576ab0696","Type":"ContainerDied","Data":"09beb8813000b7ae3422057c771c3143dc453ea5316ce1f3f8284a1b0f9a4cf9"}
Nov 02 07:59:51.255712 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.255590    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09beb8813000b7ae3422057c771c3143dc453ea5316ce1f3f8284a1b0f9a4cf9"
Nov 02 07:59:51.255712 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.255610    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"
Nov 02 07:59:51.280876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.279635    4898 scope.go:122] "RemoveContainer" containerID="3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738"
Nov 02 07:59:51.280876 ip-10-0-134-79 kubenswrapper[4898]: E1102 07:59:51.280084    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738\": container with ID starting with 3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738 not found: ID does not exist" containerID="3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738"
Nov 02 07:59:51.280876 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.280134    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738"} err="failed to get container status \"3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738\": rpc error: code = NotFound desc = could not find container \"3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738\": container with ID starting with 3f08bf2219e5d52488cf60889501db145cdbcedf7e38e36f2c29382b936ae738 not found: ID does not exist"
Nov 02 07:59:51.377592 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.377557    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"]
Nov 02 07:59:51.384287 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.384256    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-ztngm"]
Nov 02 07:59:51.469428 ip-10-0-134-79 kubenswrapper[4898]: I1102 07:59:51.469344    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe2608e-0a74-4371-9bc7-8425fb955660" path="/var/lib/kubelet/pods/dbe2608e-0a74-4371-9bc7-8425fb955660/volumes"
Nov 02 08:00:00.155479 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.155442    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"]
Nov 02 08:00:00.159537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.159504    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d8147482-0f1f-4dbe-bf62-5886c237bfe4" containerName="revoke-keys"
Nov 02 08:00:00.159537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.159538    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8147482-0f1f-4dbe-bf62-5886c237bfe4" containerName="revoke-keys"
Nov 02 08:00:00.159728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.159557    4898 cpu_manager.go:414] "containerMap: removing container" podUID="69b71e78-38e8-491a-8806-88d576ab0696" containerName="revoke-keys"
Nov 02 08:00:00.159728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.159565    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b71e78-38e8-491a-8806-88d576ab0696" containerName="revoke-keys"
Nov 02 08:00:00.159728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.159607    4898 cpu_manager.go:414] "containerMap: removing container" podUID="dbe2608e-0a74-4371-9bc7-8425fb955660" containerName="istio-proxy"
Nov 02 08:00:00.159728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.159614    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe2608e-0a74-4371-9bc7-8425fb955660" containerName="istio-proxy"
Nov 02 08:00:00.159957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.159790    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8147482-0f1f-4dbe-bf62-5886c237bfe4" containerName="revoke-keys"
Nov 02 08:00:00.159957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.159809    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b71e78-38e8-491a-8806-88d576ab0696" containerName="revoke-keys"
Nov 02 08:00:00.159957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.159827    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe2608e-0a74-4371-9bc7-8425fb955660" containerName="istio-proxy"
Nov 02 08:00:00.168907 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.165741    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"
Nov 02 08:00:00.186313 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.186280    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"]
Nov 02 08:00:00.188901 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.188824    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:00:00.205269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.205236    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:00:00.205603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.205582    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:00:00.217634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.212165    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"
Nov 02 08:00:00.224910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.220426    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7grk\" (UniqueName: \"kubernetes.io/projected/5e59499e-9863-4849-bc47-e7a91d1fcf7a-kube-api-access-f7grk\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc\" (UID: \"5e59499e-9863-4849-bc47-e7a91d1fcf7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"
Nov 02 08:00:00.224910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.220695    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e59499e-9863-4849-bc47-e7a91d1fcf7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc\" (UID: \"5e59499e-9863-4849-bc47-e7a91d1fcf7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"
Nov 02 08:00:00.224910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.220980    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"]
Nov 02 08:00:00.238951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.232168    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"
Nov 02 08:00:00.370035 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.339508    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e59499e-9863-4849-bc47-e7a91d1fcf7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc\" (UID: \"5e59499e-9863-4849-bc47-e7a91d1fcf7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"
Nov 02 08:00:00.370650 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.370622    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545\" (UID: \"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"
Nov 02 08:00:00.371084 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.371046    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8xh2\" (UniqueName: \"kubernetes.io/projected/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-kube-api-access-r8xh2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545\" (UID: \"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"
Nov 02 08:00:00.371319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.371301    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7grk\" (UniqueName: \"kubernetes.io/projected/5e59499e-9863-4849-bc47-e7a91d1fcf7a-kube-api-access-f7grk\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc\" (UID: \"5e59499e-9863-4849-bc47-e7a91d1fcf7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"
Nov 02 08:00:00.372678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.372653    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93831e27-0aad-40de-babf-ce03c15daa13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl\" (UID: \"93831e27-0aad-40de-babf-ce03c15daa13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"
Nov 02 08:00:00.372967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.372947    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmhn8\" (UniqueName: \"kubernetes.io/projected/93831e27-0aad-40de-babf-ce03c15daa13-kube-api-access-hmhn8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl\" (UID: \"93831e27-0aad-40de-babf-ce03c15daa13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"
Nov 02 08:00:00.521648 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.370326    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"]
Nov 02 08:00:00.522557 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.380505    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e59499e-9863-4849-bc47-e7a91d1fcf7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc\" (UID: \"5e59499e-9863-4849-bc47-e7a91d1fcf7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"
Nov 02 08:00:00.522557 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.519587    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7grk\" (UniqueName: \"kubernetes.io/projected/5e59499e-9863-4849-bc47-e7a91d1fcf7a-kube-api-access-f7grk\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc\" (UID: \"5e59499e-9863-4849-bc47-e7a91d1fcf7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"
Nov 02 08:00:00.545448 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.545409    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"]
Nov 02 08:00:00.548853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.548826    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"
Nov 02 08:00:00.597931 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.597497    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"]
Nov 02 08:00:00.598207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.598180    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"
Nov 02 08:00:00.606192 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.606164    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"
Nov 02 08:00:00.606470 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.606072    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"]
Nov 02 08:00:00.616319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.616298    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"]
Nov 02 08:00:00.616476 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.616462    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"]
Nov 02 08:00:00.616626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.616613    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"
Nov 02 08:00:00.617849 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.617829    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"]
Nov 02 08:00:00.623219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.623189    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4\" (UID: \"bbfacf71-eed8-4fde-96ef-c2e0c909fa30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"
Nov 02 08:00:00.623343 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.623290    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc\" (UID: \"490dc0b1-9ca5-4f40-8639-e7480c7cce8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"
Nov 02 08:00:00.623417 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.623370    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93831e27-0aad-40de-babf-ce03c15daa13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl\" (UID: \"93831e27-0aad-40de-babf-ce03c15daa13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"
Nov 02 08:00:00.623484 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.623412    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcknz\" (UniqueName: \"kubernetes.io/projected/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-kube-api-access-mcknz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4\" (UID: \"bbfacf71-eed8-4fde-96ef-c2e0c909fa30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"
Nov 02 08:00:00.623484 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.623467    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhn8\" (UniqueName: \"kubernetes.io/projected/93831e27-0aad-40de-babf-ce03c15daa13-kube-api-access-hmhn8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl\" (UID: \"93831e27-0aad-40de-babf-ce03c15daa13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"
Nov 02 08:00:00.623599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.623509    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99f50886-1073-441c-ad74-1dcc66339adc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t\" (UID: \"99f50886-1073-441c-ad74-1dcc66339adc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"
Nov 02 08:00:00.623599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.623573    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545\" (UID: \"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"
Nov 02 08:00:00.623712 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.623610    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km2bj\" (UniqueName: \"kubernetes.io/projected/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-kube-api-access-km2bj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc\" (UID: \"490dc0b1-9ca5-4f40-8639-e7480c7cce8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"
Nov 02 08:00:00.624030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.624006    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbjx\" (UniqueName: \"kubernetes.io/projected/99f50886-1073-441c-ad74-1dcc66339adc-kube-api-access-fwbjx\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t\" (UID: \"99f50886-1073-441c-ad74-1dcc66339adc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"
Nov 02 08:00:00.624172 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.624139    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8xh2\" (UniqueName: \"kubernetes.io/projected/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-kube-api-access-r8xh2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545\" (UID: \"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"
Nov 02 08:00:00.624559 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.624533    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93831e27-0aad-40de-babf-ce03c15daa13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl\" (UID: \"93831e27-0aad-40de-babf-ce03c15daa13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"
Nov 02 08:00:00.625156 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.625131    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545\" (UID: \"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"
Nov 02 08:00:00.628191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.628169    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"]
Nov 02 08:00:00.644148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.644060    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"]
Nov 02 08:00:00.657043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.657012    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"]
Nov 02 08:00:00.660817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.660785    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8xh2\" (UniqueName: \"kubernetes.io/projected/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-kube-api-access-r8xh2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545\" (UID: \"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"
Nov 02 08:00:00.665392 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.664047    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhn8\" (UniqueName: \"kubernetes.io/projected/93831e27-0aad-40de-babf-ce03c15daa13-kube-api-access-hmhn8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl\" (UID: \"93831e27-0aad-40de-babf-ce03c15daa13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"
Nov 02 08:00:00.682625 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.682582    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"]
Nov 02 08:00:00.693786 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.693543    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"
Nov 02 08:00:00.696302 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.696235    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"]
Nov 02 08:00:00.706934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.705338    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"]
Nov 02 08:00:00.711083 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.711059    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"]
Nov 02 08:00:00.711206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.711137    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"
Nov 02 08:00:00.720172 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.720068    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"]
Nov 02 08:00:00.720285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.720185    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"
Nov 02 08:00:00.720285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.720206    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"]
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.725468    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbjx\" (UniqueName: \"kubernetes.io/projected/99f50886-1073-441c-ad74-1dcc66339adc-kube-api-access-fwbjx\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t\" (UID: \"99f50886-1073-441c-ad74-1dcc66339adc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.725500    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf2ff\" (UniqueName: \"kubernetes.io/projected/516fa8a4-8039-4ca0-ac0c-83817acc91c3-kube-api-access-rf2ff\") pod \"maas-api-key-cleanup-29893440-28szk\" (UID: \"516fa8a4-8039-4ca0-ac0c-83817acc91c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.725549    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/516fa8a4-8039-4ca0-ac0c-83817acc91c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893440-28szk\" (UID: \"516fa8a4-8039-4ca0-ac0c-83817acc91c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.725616    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.725708    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx\" (UID: \"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.725805    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fwhx\" (UniqueName: \"kubernetes.io/projected/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-kube-api-access-4fwhx\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr\" (UID: \"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.725872    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r6gz\" (UniqueName: \"kubernetes.io/projected/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-kube-api-access-9r6gz\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx\" (UID: \"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.725967    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4\" (UID: \"bbfacf71-eed8-4fde-96ef-c2e0c909fa30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.726028    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr\" (UID: \"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.726073    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc\" (UID: \"490dc0b1-9ca5-4f40-8639-e7480c7cce8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.726186    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcknz\" (UniqueName: \"kubernetes.io/projected/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-kube-api-access-mcknz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4\" (UID: \"bbfacf71-eed8-4fde-96ef-c2e0c909fa30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.726249    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99f50886-1073-441c-ad74-1dcc66339adc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t\" (UID: \"99f50886-1073-441c-ad74-1dcc66339adc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.726314    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km2bj\" (UniqueName: \"kubernetes.io/projected/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-kube-api-access-km2bj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc\" (UID: \"490dc0b1-9ca5-4f40-8639-e7480c7cce8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.726846    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4\" (UID: \"bbfacf71-eed8-4fde-96ef-c2e0c909fa30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.726876    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc\" (UID: \"490dc0b1-9ca5-4f40-8639-e7480c7cce8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"
Nov 02 08:00:00.727924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.727008    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99f50886-1073-441c-ad74-1dcc66339adc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t\" (UID: \"99f50886-1073-441c-ad74-1dcc66339adc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"
Nov 02 08:00:00.729863 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.729841    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"]
Nov 02 08:00:00.733674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.733649    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"]
Nov 02 08:00:00.741318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.741292    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"]
Nov 02 08:00:00.792700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.792428    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"
Nov 02 08:00:00.794632 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.794596    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km2bj\" (UniqueName: \"kubernetes.io/projected/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-kube-api-access-km2bj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc\" (UID: \"490dc0b1-9ca5-4f40-8639-e7480c7cce8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"
Nov 02 08:00:00.794760 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.794701    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbjx\" (UniqueName: \"kubernetes.io/projected/99f50886-1073-441c-ad74-1dcc66339adc-kube-api-access-fwbjx\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t\" (UID: \"99f50886-1073-441c-ad74-1dcc66339adc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"
Nov 02 08:00:00.795157 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.795125    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcknz\" (UniqueName: \"kubernetes.io/projected/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-kube-api-access-mcknz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4\" (UID: \"bbfacf71-eed8-4fde-96ef-c2e0c909fa30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"
Nov 02 08:00:00.821926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.821696    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"
Nov 02 08:00:00.827165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.827126    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnwjj\" (UniqueName: \"kubernetes.io/projected/ed459624-7446-4717-b618-ec235eb93ce7-kube-api-access-dnwjj\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4\" (UID: \"ed459624-7446-4717-b618-ec235eb93ce7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"
Nov 02 08:00:00.827281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.827165    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed459624-7446-4717-b618-ec235eb93ce7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4\" (UID: \"ed459624-7446-4717-b618-ec235eb93ce7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"
Nov 02 08:00:00.827281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.827209    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/516fa8a4-8039-4ca0-ac0c-83817acc91c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893440-28szk\" (UID: \"516fa8a4-8039-4ca0-ac0c-83817acc91c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"
Nov 02 08:00:00.827281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.827256    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx\" (UID: \"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"
Nov 02 08:00:00.827467 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.827297    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fwhx\" (UniqueName: \"kubernetes.io/projected/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-kube-api-access-4fwhx\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr\" (UID: \"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"
Nov 02 08:00:00.827467 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.827327    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgh5z\" (UniqueName: \"kubernetes.io/projected/42866b28-00e6-4781-a563-b42446b3e432-kube-api-access-mgh5z\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5\" (UID: \"42866b28-00e6-4781-a563-b42446b3e432\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"
Nov 02 08:00:00.827467 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.827359    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r6gz\" (UniqueName: \"kubernetes.io/projected/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-kube-api-access-9r6gz\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx\" (UID: \"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"
Nov 02 08:00:00.827467 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.827429    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr\" (UID: \"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"
Nov 02 08:00:00.828248 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.828223    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/516fa8a4-8039-4ca0-ac0c-83817acc91c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893440-28szk\" (UID: \"516fa8a4-8039-4ca0-ac0c-83817acc91c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"
Nov 02 08:00:00.828366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.828283    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42866b28-00e6-4781-a563-b42446b3e432-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5\" (UID: \"42866b28-00e6-4781-a563-b42446b3e432\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"
Nov 02 08:00:00.828432 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.828399    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf2ff\" (UniqueName: \"kubernetes.io/projected/516fa8a4-8039-4ca0-ac0c-83817acc91c3-kube-api-access-rf2ff\") pod \"maas-api-key-cleanup-29893440-28szk\" (UID: \"516fa8a4-8039-4ca0-ac0c-83817acc91c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"
Nov 02 08:00:00.828792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.828561    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx\" (UID: \"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"
Nov 02 08:00:00.829222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.829187    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr\" (UID: \"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"
Nov 02 08:00:00.851134 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.851097    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fwhx\" (UniqueName: \"kubernetes.io/projected/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-kube-api-access-4fwhx\") pod \"maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr\" (UID: \"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"
Nov 02 08:00:00.852059 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.852037    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r6gz\" (UniqueName: \"kubernetes.io/projected/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-kube-api-access-9r6gz\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx\" (UID: \"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"
Nov 02 08:00:00.865310 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.863032    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf2ff\" (UniqueName: \"kubernetes.io/projected/516fa8a4-8039-4ca0-ac0c-83817acc91c3-kube-api-access-rf2ff\") pod \"maas-api-key-cleanup-29893440-28szk\" (UID: \"516fa8a4-8039-4ca0-ac0c-83817acc91c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"
Nov 02 08:00:00.912213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.911039    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"
Nov 02 08:00:00.917206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.916501    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"
Nov 02 08:00:00.943280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.929770    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42866b28-00e6-4781-a563-b42446b3e432-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5\" (UID: \"42866b28-00e6-4781-a563-b42446b3e432\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"
Nov 02 08:00:00.943280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.929958    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnwjj\" (UniqueName: \"kubernetes.io/projected/ed459624-7446-4717-b618-ec235eb93ce7-kube-api-access-dnwjj\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4\" (UID: \"ed459624-7446-4717-b618-ec235eb93ce7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"
Nov 02 08:00:00.943280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.929991    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed459624-7446-4717-b618-ec235eb93ce7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4\" (UID: \"ed459624-7446-4717-b618-ec235eb93ce7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"
Nov 02 08:00:00.943280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.930072    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgh5z\" (UniqueName: \"kubernetes.io/projected/42866b28-00e6-4781-a563-b42446b3e432-kube-api-access-mgh5z\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5\" (UID: \"42866b28-00e6-4781-a563-b42446b3e432\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"
Nov 02 08:00:00.943280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.931195    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42866b28-00e6-4781-a563-b42446b3e432-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5\" (UID: \"42866b28-00e6-4781-a563-b42446b3e432\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"
Nov 02 08:00:00.943280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.931777    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed459624-7446-4717-b618-ec235eb93ce7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4\" (UID: \"ed459624-7446-4717-b618-ec235eb93ce7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"
Nov 02 08:00:00.943280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.932061    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"
Nov 02 08:00:00.947002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.945025    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"
Nov 02 08:00:00.992346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.971388    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgh5z\" (UniqueName: \"kubernetes.io/projected/42866b28-00e6-4781-a563-b42446b3e432-kube-api-access-mgh5z\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5\" (UID: \"42866b28-00e6-4781-a563-b42446b3e432\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"
Nov 02 08:00:01.002934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:00.999139    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"
Nov 02 08:00:01.008122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:01.008092    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnwjj\" (UniqueName: \"kubernetes.io/projected/ed459624-7446-4717-b618-ec235eb93ce7-kube-api-access-dnwjj\") pod \"maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4\" (UID: \"ed459624-7446-4717-b618-ec235eb93ce7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"
Nov 02 08:00:01.013569 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:01.013530    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"
Nov 02 08:00:01.028544 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:01.028340    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"
Nov 02 08:00:01.039678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:01.039647    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"
Nov 02 08:00:01.046369 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:01.046341    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"
Nov 02 08:00:01.252644 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:00:01.252613    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e59499e_9863_4849_bc47_e7a91d1fcf7a.slice/crio-423b403f0dbccaf05f578310ed6495f363d6e16a06e6917f6ea8f789bf362472 WatchSource:0}: Error finding container 423b403f0dbccaf05f578310ed6495f363d6e16a06e6917f6ea8f789bf362472: Status 404 returned error can't find the container with id 423b403f0dbccaf05f578310ed6495f363d6e16a06e6917f6ea8f789bf362472
Nov 02 08:00:01.299203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:01.294359    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"]
Nov 02 08:00:01.299824 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:00:01.299401    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dd6fad_2ea4_499e_9b3c_0f65fad9cedb.slice/crio-899aac9cc80172fafd601ad0f063ade63099161876ee75e388249461dc7fcea1 WatchSource:0}: Error finding container 899aac9cc80172fafd601ad0f063ade63099161876ee75e388249461dc7fcea1: Status 404 returned error can't find the container with id 899aac9cc80172fafd601ad0f063ade63099161876ee75e388249461dc7fcea1
Nov 02 08:00:01.323284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:01.321317    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"]
Nov 02 08:00:01.698542 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:01.698249    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc" event={"ID":"5e59499e-9863-4849-bc47-e7a91d1fcf7a","Type":"ContainerStarted","Data":"423b403f0dbccaf05f578310ed6495f363d6e16a06e6917f6ea8f789bf362472"}
Nov 02 08:00:01.701188 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:01.700824    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545" event={"ID":"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb","Type":"ContainerStarted","Data":"899aac9cc80172fafd601ad0f063ade63099161876ee75e388249461dc7fcea1"}
Nov 02 08:00:01.715793 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:01.715703    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"]
Nov 02 08:00:01.721813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:01.721781    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"]
Nov 02 08:00:01.892669 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:00:01.892593    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93831e27_0aad_40de_babf_ce03c15daa13.slice/crio-29648d352321e5b27815144930e13bd86ad34a49bb0d3244cf262609e1c16de7 WatchSource:0}: Error finding container 29648d352321e5b27815144930e13bd86ad34a49bb0d3244cf262609e1c16de7: Status 404 returned error can't find the container with id 29648d352321e5b27815144930e13bd86ad34a49bb0d3244cf262609e1c16de7
Nov 02 08:00:02.001105 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.000972    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"]
Nov 02 08:00:02.008076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.007978    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"]
Nov 02 08:00:02.014522 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.014347    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"]
Nov 02 08:00:02.100613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.100587    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"]
Nov 02 08:00:02.133949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.133921    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"]
Nov 02 08:00:02.195046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.195017    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"]
Nov 02 08:00:02.205884 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.205856    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"]
Nov 02 08:00:02.276926 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:00:02.274440    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92df2ddf_95bc_4fbd_ada9_df92a5ca62cd.slice/crio-81192508cd816bb49c66e06d2c9ad6a67b4ba7709104ce27875a76507b2471ba WatchSource:0}: Error finding container 81192508cd816bb49c66e06d2c9ad6a67b4ba7709104ce27875a76507b2471ba: Status 404 returned error can't find the container with id 81192508cd816bb49c66e06d2c9ad6a67b4ba7709104ce27875a76507b2471ba
Nov 02 08:00:02.712366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.712332    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl" event={"ID":"93831e27-0aad-40de-babf-ce03c15daa13","Type":"ContainerStarted","Data":"fdb8b9dd269e4944ff601b7dd45270ca91fead73a0535fc89fd70a72fec98cef"}
Nov 02 08:00:02.712649 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.712395    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl" event={"ID":"93831e27-0aad-40de-babf-ce03c15daa13","Type":"ContainerStarted","Data":"29648d352321e5b27815144930e13bd86ad34a49bb0d3244cf262609e1c16de7"}
Nov 02 08:00:02.713914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.713877    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx" event={"ID":"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd","Type":"ContainerStarted","Data":"c934da84132fb1f2cd826e32df0988f42154bc605f2fce06bae56980d0537419"}
Nov 02 08:00:02.715200 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.715172    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc" event={"ID":"490dc0b1-9ca5-4f40-8639-e7480c7cce8d","Type":"ContainerStarted","Data":"d1ace0b2217f423b437e3c27773f84f110dfbe15f290d09f9a4e3ba83df41aee"}
Nov 02 08:00:02.716954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.716924    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t" event={"ID":"99f50886-1073-441c-ad74-1dcc66339adc","Type":"ContainerStarted","Data":"b96638da4f20a7269f35d861ce35d885394c8e95277af8f1017df0eb87c8541b"}
Nov 02 08:00:02.717067 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.716977    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t" event={"ID":"99f50886-1073-441c-ad74-1dcc66339adc","Type":"ContainerStarted","Data":"a5204dc1a94d30311f537236f9365cbf5fd47421b89cfb05d83570692fafcbf1"}
Nov 02 08:00:02.718350 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.718330    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc" event={"ID":"5e59499e-9863-4849-bc47-e7a91d1fcf7a","Type":"ContainerStarted","Data":"f3cd404803b953e01f4a71d3531aeb251ec16dc08d6c4fcff63b9c8bd2791f4a"}
Nov 02 08:00:02.720363 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.720340    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4" event={"ID":"ed459624-7446-4717-b618-ec235eb93ce7","Type":"ContainerStarted","Data":"ebc6e686b3e059b909f63ecdc6549f2cee1557d421453298be28893db669859c"}
Nov 02 08:00:02.720450 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.720386    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4" event={"ID":"ed459624-7446-4717-b618-ec235eb93ce7","Type":"ContainerStarted","Data":"b7af7b4c3dbbbda70f3249293e957ed5a3a13453d4bd84ac3a8bffe0b7485b26"}
Nov 02 08:00:02.721856 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.721815    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk" event={"ID":"516fa8a4-8039-4ca0-ac0c-83817acc91c3","Type":"ContainerStarted","Data":"97667419c7dc68569cff132309cd006bbc7b0cc7b038ced35285f346390eef92"}
Nov 02 08:00:02.723363 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.723343    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545" event={"ID":"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb","Type":"ContainerStarted","Data":"bfb64832b5dead644813d7e2dc111902aa63857520420485d6206d73af10ee44"}
Nov 02 08:00:02.724618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.724598    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5" event={"ID":"42866b28-00e6-4781-a563-b42446b3e432","Type":"ContainerStarted","Data":"43b453274512719cfa0e0d8eff56e6135fc632ccc8c8b985102826e50b503159"}
Nov 02 08:00:02.725903 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.725871    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr" event={"ID":"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd","Type":"ContainerStarted","Data":"81192508cd816bb49c66e06d2c9ad6a67b4ba7709104ce27875a76507b2471ba"}
Nov 02 08:00:02.727320 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.727300    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4" event={"ID":"bbfacf71-eed8-4fde-96ef-c2e0c909fa30","Type":"ContainerStarted","Data":"51bb8374329b4fe641cdeaa1e8ed4c00a0938567502732d73e541c4aff757887"}
Nov 02 08:00:02.727402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.727334    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4" event={"ID":"bbfacf71-eed8-4fde-96ef-c2e0c909fa30","Type":"ContainerStarted","Data":"49695756af242a65c9e7f11e1d80aaabc29fda99a9bb95dc38f83090f9b65f64"}
Nov 02 08:00:02.759865 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.756846    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl" podStartSLOduration=2.756830051 podStartE2EDuration="2.756830051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:02.754497475 +0000 UTC m=+2153.444824440" watchObservedRunningTime="2026-11-02 08:00:02.756830051 +0000 UTC m=+2153.447157016"
Nov 02 08:00:02.804484 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.804415    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc" podStartSLOduration=2.804394329 podStartE2EDuration="2.804394329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:02.8018238 +0000 UTC m=+2153.492150764" watchObservedRunningTime="2026-11-02 08:00:02.804394329 +0000 UTC m=+2153.494721293"
Nov 02 08:00:02.845741 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.845683    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4" podStartSLOduration=2.845664452 podStartE2EDuration="2.845664452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:02.844975407 +0000 UTC m=+2153.535302390" watchObservedRunningTime="2026-11-02 08:00:02.845664452 +0000 UTC m=+2153.535991421"
Nov 02 08:00:02.899953 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.899880    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t" podStartSLOduration=2.89985882 podStartE2EDuration="2.89985882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:02.898298814 +0000 UTC m=+2153.588625779" watchObservedRunningTime="2026-11-02 08:00:02.89985882 +0000 UTC m=+2153.590185785"
Nov 02 08:00:02.954527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:02.954450    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4" podStartSLOduration=2.9544285390000002 podStartE2EDuration="2.954428539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:02.940872378 +0000 UTC m=+2153.631199342" watchObservedRunningTime="2026-11-02 08:00:02.954428539 +0000 UTC m=+2153.644755504"
Nov 02 08:00:03.005726 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.005677    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545" podStartSLOduration=3.005663663 podStartE2EDuration="3.005663663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:03.00031507 +0000 UTC m=+2153.690642059" watchObservedRunningTime="2026-11-02 08:00:03.005663663 +0000 UTC m=+2153.695990646"
Nov 02 08:00:03.745323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.745232    4898 generic.go:356] "Generic (PLEG): container finished" podID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" containerID="ea4517b48f4a91f39d9f15bf6503780739bb9790d1e7dc5ec1ee89fd41391362" exitCode=60
Nov 02 08:00:03.745803 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.745319    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx" event={"ID":"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd","Type":"ContainerDied","Data":"ea4517b48f4a91f39d9f15bf6503780739bb9790d1e7dc5ec1ee89fd41391362"}
Nov 02 08:00:03.745969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.745539    4898 scope.go:122] "RemoveContainer" containerID="ea4517b48f4a91f39d9f15bf6503780739bb9790d1e7dc5ec1ee89fd41391362"
Nov 02 08:00:03.747444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.747419    4898 generic.go:356] "Generic (PLEG): container finished" podID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" containerID="950df5971c0810f94c111b08a679a86fee699051dc64bd2f5537d5e387ae034b" exitCode=60
Nov 02 08:00:03.747603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.747544    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc" event={"ID":"490dc0b1-9ca5-4f40-8639-e7480c7cce8d","Type":"ContainerDied","Data":"950df5971c0810f94c111b08a679a86fee699051dc64bd2f5537d5e387ae034b"}
Nov 02 08:00:03.747665 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.747649    4898 scope.go:122] "RemoveContainer" containerID="950df5971c0810f94c111b08a679a86fee699051dc64bd2f5537d5e387ae034b"
Nov 02 08:00:03.750949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.750843    4898 generic.go:356] "Generic (PLEG): container finished" podID="99f50886-1073-441c-ad74-1dcc66339adc" containerID="b96638da4f20a7269f35d861ce35d885394c8e95277af8f1017df0eb87c8541b" exitCode=60
Nov 02 08:00:03.750949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.750915    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t" event={"ID":"99f50886-1073-441c-ad74-1dcc66339adc","Type":"ContainerDied","Data":"b96638da4f20a7269f35d861ce35d885394c8e95277af8f1017df0eb87c8541b"}
Nov 02 08:00:03.751175 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.751156    4898 scope.go:122] "RemoveContainer" containerID="b96638da4f20a7269f35d861ce35d885394c8e95277af8f1017df0eb87c8541b"
Nov 02 08:00:03.752954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.752931    4898 generic.go:356] "Generic (PLEG): container finished" podID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" containerID="f3cd404803b953e01f4a71d3531aeb251ec16dc08d6c4fcff63b9c8bd2791f4a" exitCode=60
Nov 02 08:00:03.753051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.753006    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc" event={"ID":"5e59499e-9863-4849-bc47-e7a91d1fcf7a","Type":"ContainerDied","Data":"f3cd404803b953e01f4a71d3531aeb251ec16dc08d6c4fcff63b9c8bd2791f4a"}
Nov 02 08:00:03.753160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.753144    4898 scope.go:122] "RemoveContainer" containerID="f3cd404803b953e01f4a71d3531aeb251ec16dc08d6c4fcff63b9c8bd2791f4a"
Nov 02 08:00:03.754874 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.754847    4898 generic.go:356] "Generic (PLEG): container finished" podID="ed459624-7446-4717-b618-ec235eb93ce7" containerID="ebc6e686b3e059b909f63ecdc6549f2cee1557d421453298be28893db669859c" exitCode=60
Nov 02 08:00:03.769168 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.755014    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4" event={"ID":"ed459624-7446-4717-b618-ec235eb93ce7","Type":"ContainerDied","Data":"ebc6e686b3e059b909f63ecdc6549f2cee1557d421453298be28893db669859c"}
Nov 02 08:00:03.769168 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.755136    4898 scope.go:122] "RemoveContainer" containerID="ebc6e686b3e059b909f63ecdc6549f2cee1557d421453298be28893db669859c"
Nov 02 08:00:03.813312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.812757    4898 generic.go:356] "Generic (PLEG): container finished" podID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" containerID="1b129d89bac76138a94479e38b0518d99c21cece85f7402552a869c9362c86f7" exitCode=60
Nov 02 08:00:03.813312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.812860    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk" event={"ID":"516fa8a4-8039-4ca0-ac0c-83817acc91c3","Type":"ContainerDied","Data":"1b129d89bac76138a94479e38b0518d99c21cece85f7402552a869c9362c86f7"}
Nov 02 08:00:03.813312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.813151    4898 scope.go:122] "RemoveContainer" containerID="1b129d89bac76138a94479e38b0518d99c21cece85f7402552a869c9362c86f7"
Nov 02 08:00:03.816105 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.816081    4898 generic.go:356] "Generic (PLEG): container finished" podID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" containerID="bfb64832b5dead644813d7e2dc111902aa63857520420485d6206d73af10ee44" exitCode=60
Nov 02 08:00:03.816224 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.816161    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545" event={"ID":"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb","Type":"ContainerDied","Data":"bfb64832b5dead644813d7e2dc111902aa63857520420485d6206d73af10ee44"}
Nov 02 08:00:03.816404 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.816387    4898 scope.go:122] "RemoveContainer" containerID="bfb64832b5dead644813d7e2dc111902aa63857520420485d6206d73af10ee44"
Nov 02 08:00:03.818123 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.818103    4898 generic.go:356] "Generic (PLEG): container finished" podID="42866b28-00e6-4781-a563-b42446b3e432" containerID="ca62f6d8f14d4f3bd22f5b8e02d33c727695ff85494193f9c49cb6c27f11cdf6" exitCode=60
Nov 02 08:00:03.818215 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.818164    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5" event={"ID":"42866b28-00e6-4781-a563-b42446b3e432","Type":"ContainerDied","Data":"ca62f6d8f14d4f3bd22f5b8e02d33c727695ff85494193f9c49cb6c27f11cdf6"}
Nov 02 08:00:03.818421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.818407    4898 scope.go:122] "RemoveContainer" containerID="ca62f6d8f14d4f3bd22f5b8e02d33c727695ff85494193f9c49cb6c27f11cdf6"
Nov 02 08:00:03.820635 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.820611    4898 generic.go:356] "Generic (PLEG): container finished" podID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" containerID="5199cbfd064867fe4633e7cd00d1b86d5f103297b60572216700f7c52064c9df" exitCode=60
Nov 02 08:00:03.820740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.820724    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr" event={"ID":"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd","Type":"ContainerDied","Data":"5199cbfd064867fe4633e7cd00d1b86d5f103297b60572216700f7c52064c9df"}
Nov 02 08:00:03.821091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.821072    4898 scope.go:122] "RemoveContainer" containerID="5199cbfd064867fe4633e7cd00d1b86d5f103297b60572216700f7c52064c9df"
Nov 02 08:00:03.825041 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.824507    4898 generic.go:356] "Generic (PLEG): container finished" podID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" containerID="51bb8374329b4fe641cdeaa1e8ed4c00a0938567502732d73e541c4aff757887" exitCode=60
Nov 02 08:00:03.825041 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.824610    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4" event={"ID":"bbfacf71-eed8-4fde-96ef-c2e0c909fa30","Type":"ContainerDied","Data":"51bb8374329b4fe641cdeaa1e8ed4c00a0938567502732d73e541c4aff757887"}
Nov 02 08:00:03.825041 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.824921    4898 scope.go:122] "RemoveContainer" containerID="51bb8374329b4fe641cdeaa1e8ed4c00a0938567502732d73e541c4aff757887"
Nov 02 08:00:03.827950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.826699    4898 generic.go:356] "Generic (PLEG): container finished" podID="93831e27-0aad-40de-babf-ce03c15daa13" containerID="fdb8b9dd269e4944ff601b7dd45270ca91fead73a0535fc89fd70a72fec98cef" exitCode=60
Nov 02 08:00:03.827950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.826754    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl" event={"ID":"93831e27-0aad-40de-babf-ce03c15daa13","Type":"ContainerDied","Data":"fdb8b9dd269e4944ff601b7dd45270ca91fead73a0535fc89fd70a72fec98cef"}
Nov 02 08:00:03.827950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:03.827368    4898 scope.go:122] "RemoveContainer" containerID="fdb8b9dd269e4944ff601b7dd45270ca91fead73a0535fc89fd70a72fec98cef"
Nov 02 08:00:04.838149 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:04.838104    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk" event={"ID":"516fa8a4-8039-4ca0-ac0c-83817acc91c3","Type":"ContainerStarted","Data":"ffc7ead0678f2958118d08d417591f833139db1d153cb82336a0d00b433363b9"}
Nov 02 08:00:04.839969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:04.839910    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545" event={"ID":"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb","Type":"ContainerStarted","Data":"81bd1faf86f5ae1ee2a8be7e4b03cc44197958469ae3a030885cb83acd5c3d5f"}
Nov 02 08:00:04.841595 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:04.841573    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5" event={"ID":"42866b28-00e6-4781-a563-b42446b3e432","Type":"ContainerStarted","Data":"bd5f01825ddea9dd42858522d8574da79dbb20655c716d3f75e2255b2d809334"}
Nov 02 08:00:04.848926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:04.844968    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr" event={"ID":"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd","Type":"ContainerStarted","Data":"6a3b4ba8e9ebd38cce57f93f55d3c123843bb54e4b9349c24b792248ea84d0eb"}
Nov 02 08:00:04.852165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:04.852105    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4" event={"ID":"bbfacf71-eed8-4fde-96ef-c2e0c909fa30","Type":"ContainerStarted","Data":"0a9205c5ce4dbdfbfc33b3d7c451951e26001a9c0d4b4eb5bf3e19fb4a89aaaa"}
Nov 02 08:00:04.854543 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:04.854519    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx" event={"ID":"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd","Type":"ContainerStarted","Data":"99c588b6eb4b621de2616e3ad3d297e52fdfd063229db6f37fde266d7575a30f"}
Nov 02 08:00:04.909412 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:04.908554    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc" event={"ID":"490dc0b1-9ca5-4f40-8639-e7480c7cce8d","Type":"ContainerStarted","Data":"9975ffee1050a60e6e449c5e67c44fa112f03ef35b0e0582e947218286ffd9b6"}
Nov 02 08:00:04.911854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:04.911723    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk" podStartSLOduration=4.911705397 podStartE2EDuration="4.911705397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:04.907198266 +0000 UTC m=+2155.597525231" watchObservedRunningTime="2026-11-02 08:00:04.911705397 +0000 UTC m=+2155.602032363"
Nov 02 08:00:04.913556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:04.912956    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t" event={"ID":"99f50886-1073-441c-ad74-1dcc66339adc","Type":"ContainerStarted","Data":"b118ab2224d983ad7b15c61bf5387b9489a9e1f78d2ed30cabb413706bc9b631"}
Nov 02 08:00:04.916184 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:04.916162    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc" event={"ID":"5e59499e-9863-4849-bc47-e7a91d1fcf7a","Type":"ContainerStarted","Data":"599be23846f72271af112cb2a6d4f01ea5068f74d29bc87ad62cc6b949c2e37e"}
Nov 02 08:00:04.919509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:04.919022    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4" event={"ID":"ed459624-7446-4717-b618-ec235eb93ce7","Type":"ContainerStarted","Data":"a3f7929593ecebe830f4fa9c72e2675cd95c27590110f2243ee3a8d7bb9294ad"}
Nov 02 08:00:05.013514 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.013447    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5" podStartSLOduration=5.013426173 podStartE2EDuration="5.013426173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:05.004440992 +0000 UTC m=+2155.694767959" watchObservedRunningTime="2026-11-02 08:00:05.013426173 +0000 UTC m=+2155.703753138"
Nov 02 08:00:05.076915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.056084    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx" podStartSLOduration=5.056063816 podStartE2EDuration="5.056063816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:05.053030578 +0000 UTC m=+2155.743357545" watchObservedRunningTime="2026-11-02 08:00:05.056063816 +0000 UTC m=+2155.746390782"
Nov 02 08:00:05.223936 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.223857    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr" podStartSLOduration=5.223843788 podStartE2EDuration="5.223843788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:05.222218742 +0000 UTC m=+2155.912545734" watchObservedRunningTime="2026-11-02 08:00:05.223843788 +0000 UTC m=+2155.914170763"
Nov 02 08:00:05.444443 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.444391    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc" podStartSLOduration=5.444377008 podStartE2EDuration="5.444377008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:05.439676766 +0000 UTC m=+2156.130003729" watchObservedRunningTime="2026-11-02 08:00:05.444377008 +0000 UTC m=+2156.134703971"
Nov 02 08:00:05.931089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.931054    4898 generic.go:356] "Generic (PLEG): container finished" podID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" containerID="99c588b6eb4b621de2616e3ad3d297e52fdfd063229db6f37fde266d7575a30f" exitCode=60
Nov 02 08:00:05.931573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.931144    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx" event={"ID":"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd","Type":"ContainerDied","Data":"99c588b6eb4b621de2616e3ad3d297e52fdfd063229db6f37fde266d7575a30f"}
Nov 02 08:00:05.931573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.931258    4898 scope.go:122] "RemoveContainer" containerID="ea4517b48f4a91f39d9f15bf6503780739bb9790d1e7dc5ec1ee89fd41391362"
Nov 02 08:00:05.931573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.931453    4898 scope.go:122] "RemoveContainer" containerID="99c588b6eb4b621de2616e3ad3d297e52fdfd063229db6f37fde266d7575a30f"
Nov 02 08:00:05.931862 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:05.931835    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx_odh-ai-gateway-infra(ce754bc4-4ab3-4b5b-8cb0-d35d080413bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd"
Nov 02 08:00:05.933410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.933388    4898 generic.go:356] "Generic (PLEG): container finished" podID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" containerID="9975ffee1050a60e6e449c5e67c44fa112f03ef35b0e0582e947218286ffd9b6" exitCode=60
Nov 02 08:00:05.933520 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.933461    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc" event={"ID":"490dc0b1-9ca5-4f40-8639-e7480c7cce8d","Type":"ContainerDied","Data":"9975ffee1050a60e6e449c5e67c44fa112f03ef35b0e0582e947218286ffd9b6"}
Nov 02 08:00:05.933719 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.933704    4898 scope.go:122] "RemoveContainer" containerID="9975ffee1050a60e6e449c5e67c44fa112f03ef35b0e0582e947218286ffd9b6"
Nov 02 08:00:05.934041 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:05.934017    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc_odh-ai-gateway-infra(490dc0b1-9ca5-4f40-8639-e7480c7cce8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d"
Nov 02 08:00:05.935596 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.935573    4898 generic.go:356] "Generic (PLEG): container finished" podID="99f50886-1073-441c-ad74-1dcc66339adc" containerID="b118ab2224d983ad7b15c61bf5387b9489a9e1f78d2ed30cabb413706bc9b631" exitCode=60
Nov 02 08:00:05.935712 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.935657    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t" event={"ID":"99f50886-1073-441c-ad74-1dcc66339adc","Type":"ContainerDied","Data":"b118ab2224d983ad7b15c61bf5387b9489a9e1f78d2ed30cabb413706bc9b631"}
Nov 02 08:00:05.935952 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.935930    4898 scope.go:122] "RemoveContainer" containerID="b118ab2224d983ad7b15c61bf5387b9489a9e1f78d2ed30cabb413706bc9b631"
Nov 02 08:00:05.936246 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:05.936212    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t_odh-ai-gateway-infra(99f50886-1073-441c-ad74-1dcc66339adc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t" podUID="99f50886-1073-441c-ad74-1dcc66339adc"
Nov 02 08:00:05.938142 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.938120    4898 generic.go:356] "Generic (PLEG): container finished" podID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" containerID="599be23846f72271af112cb2a6d4f01ea5068f74d29bc87ad62cc6b949c2e37e" exitCode=60
Nov 02 08:00:05.938247 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.938197    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc" event={"ID":"5e59499e-9863-4849-bc47-e7a91d1fcf7a","Type":"ContainerDied","Data":"599be23846f72271af112cb2a6d4f01ea5068f74d29bc87ad62cc6b949c2e37e"}
Nov 02 08:00:05.938373 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.938360    4898 scope.go:122] "RemoveContainer" containerID="599be23846f72271af112cb2a6d4f01ea5068f74d29bc87ad62cc6b949c2e37e"
Nov 02 08:00:05.938556 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:05.938538    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc_odh-ai-gateway-infra(5e59499e-9863-4849-bc47-e7a91d1fcf7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a"
Nov 02 08:00:05.940461 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.940438    4898 generic.go:356] "Generic (PLEG): container finished" podID="ed459624-7446-4717-b618-ec235eb93ce7" containerID="a3f7929593ecebe830f4fa9c72e2675cd95c27590110f2243ee3a8d7bb9294ad" exitCode=60
Nov 02 08:00:05.940558 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.940521    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4" event={"ID":"ed459624-7446-4717-b618-ec235eb93ce7","Type":"ContainerDied","Data":"a3f7929593ecebe830f4fa9c72e2675cd95c27590110f2243ee3a8d7bb9294ad"}
Nov 02 08:00:05.940693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.940678    4898 scope.go:122] "RemoveContainer" containerID="a3f7929593ecebe830f4fa9c72e2675cd95c27590110f2243ee3a8d7bb9294ad"
Nov 02 08:00:05.940922 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:05.940868    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4_odh-ai-gateway-infra(ed459624-7446-4717-b618-ec235eb93ce7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4" podUID="ed459624-7446-4717-b618-ec235eb93ce7"
Nov 02 08:00:05.942640 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.942618    4898 generic.go:356] "Generic (PLEG): container finished" podID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" containerID="ffc7ead0678f2958118d08d417591f833139db1d153cb82336a0d00b433363b9" exitCode=60
Nov 02 08:00:05.942738 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.942680    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk" event={"ID":"516fa8a4-8039-4ca0-ac0c-83817acc91c3","Type":"ContainerDied","Data":"ffc7ead0678f2958118d08d417591f833139db1d153cb82336a0d00b433363b9"}
Nov 02 08:00:05.942967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.942951    4898 scope.go:122] "RemoveContainer" containerID="ffc7ead0678f2958118d08d417591f833139db1d153cb82336a0d00b433363b9"
Nov 02 08:00:05.943182 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:05.943161    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893440-28szk_odh-ai-gateway-infra(516fa8a4-8039-4ca0-ac0c-83817acc91c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3"
Nov 02 08:00:05.944848 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.944829    4898 generic.go:356] "Generic (PLEG): container finished" podID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" containerID="81bd1faf86f5ae1ee2a8be7e4b03cc44197958469ae3a030885cb83acd5c3d5f" exitCode=60
Nov 02 08:00:05.944975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.944905    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545" event={"ID":"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb","Type":"ContainerDied","Data":"81bd1faf86f5ae1ee2a8be7e4b03cc44197958469ae3a030885cb83acd5c3d5f"}
Nov 02 08:00:05.945101 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.945083    4898 scope.go:122] "RemoveContainer" containerID="81bd1faf86f5ae1ee2a8be7e4b03cc44197958469ae3a030885cb83acd5c3d5f"
Nov 02 08:00:05.945302 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:05.945284    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545_odh-ai-gateway-infra(f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb"
Nov 02 08:00:05.946470 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.946454    4898 scope.go:122] "RemoveContainer" containerID="950df5971c0810f94c111b08a679a86fee699051dc64bd2f5537d5e387ae034b"
Nov 02 08:00:05.947145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.947122    4898 generic.go:356] "Generic (PLEG): container finished" podID="42866b28-00e6-4781-a563-b42446b3e432" containerID="bd5f01825ddea9dd42858522d8574da79dbb20655c716d3f75e2255b2d809334" exitCode=60
Nov 02 08:00:05.947256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.947150    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5" event={"ID":"42866b28-00e6-4781-a563-b42446b3e432","Type":"ContainerDied","Data":"bd5f01825ddea9dd42858522d8574da79dbb20655c716d3f75e2255b2d809334"}
Nov 02 08:00:05.947504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.947420    4898 scope.go:122] "RemoveContainer" containerID="bd5f01825ddea9dd42858522d8574da79dbb20655c716d3f75e2255b2d809334"
Nov 02 08:00:05.947658 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:05.947637    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5_odh-ai-gateway-infra(42866b28-00e6-4781-a563-b42446b3e432)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5" podUID="42866b28-00e6-4781-a563-b42446b3e432"
Nov 02 08:00:05.949264 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.949243    4898 generic.go:356] "Generic (PLEG): container finished" podID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" containerID="6a3b4ba8e9ebd38cce57f93f55d3c123843bb54e4b9349c24b792248ea84d0eb" exitCode=60
Nov 02 08:00:05.949381 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.949329    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr" event={"ID":"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd","Type":"ContainerDied","Data":"6a3b4ba8e9ebd38cce57f93f55d3c123843bb54e4b9349c24b792248ea84d0eb"}
Nov 02 08:00:05.949646 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.949624    4898 scope.go:122] "RemoveContainer" containerID="6a3b4ba8e9ebd38cce57f93f55d3c123843bb54e4b9349c24b792248ea84d0eb"
Nov 02 08:00:05.949866 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:05.949844    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr_odh-ai-gateway-infra(92df2ddf-95bc-4fbd-ada9-df92a5ca62cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd"
Nov 02 08:00:05.951599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.951576    4898 generic.go:356] "Generic (PLEG): container finished" podID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" containerID="0a9205c5ce4dbdfbfc33b3d7c451951e26001a9c0d4b4eb5bf3e19fb4a89aaaa" exitCode=60
Nov 02 08:00:05.951696 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.951605    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4" event={"ID":"bbfacf71-eed8-4fde-96ef-c2e0c909fa30","Type":"ContainerDied","Data":"0a9205c5ce4dbdfbfc33b3d7c451951e26001a9c0d4b4eb5bf3e19fb4a89aaaa"}
Nov 02 08:00:05.951876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.951859    4898 scope.go:122] "RemoveContainer" containerID="0a9205c5ce4dbdfbfc33b3d7c451951e26001a9c0d4b4eb5bf3e19fb4a89aaaa"
Nov 02 08:00:05.952096 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:05.952074    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4_odh-ai-gateway-infra(bbfacf71-eed8-4fde-96ef-c2e0c909fa30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30"
Nov 02 08:00:05.953745 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.953723    4898 generic.go:356] "Generic (PLEG): container finished" podID="93831e27-0aad-40de-babf-ce03c15daa13" containerID="6097a354ae1810e78ad4af1c38dec2c5cea23a4dfd423dea843c3c8b95ceb278" exitCode=60
Nov 02 08:00:05.953822 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.953777    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl" event={"ID":"93831e27-0aad-40de-babf-ce03c15daa13","Type":"ContainerDied","Data":"6097a354ae1810e78ad4af1c38dec2c5cea23a4dfd423dea843c3c8b95ceb278"}
Nov 02 08:00:05.954058 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.954042    4898 scope.go:122] "RemoveContainer" containerID="6097a354ae1810e78ad4af1c38dec2c5cea23a4dfd423dea843c3c8b95ceb278"
Nov 02 08:00:05.954302 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:05.954283    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl_odh-ai-gateway-infra(93831e27-0aad-40de-babf-ce03c15daa13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl" podUID="93831e27-0aad-40de-babf-ce03c15daa13"
Nov 02 08:00:05.962069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.962037    4898 scope.go:122] "RemoveContainer" containerID="b96638da4f20a7269f35d861ce35d885394c8e95277af8f1017df0eb87c8541b"
Nov 02 08:00:05.983061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:05.981507    4898 scope.go:122] "RemoveContainer" containerID="f3cd404803b953e01f4a71d3531aeb251ec16dc08d6c4fcff63b9c8bd2791f4a"
Nov 02 08:00:06.009164 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:06.009141    4898 scope.go:122] "RemoveContainer" containerID="ebc6e686b3e059b909f63ecdc6549f2cee1557d421453298be28893db669859c"
Nov 02 08:00:06.025961 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:06.025939    4898 scope.go:122] "RemoveContainer" containerID="1b129d89bac76138a94479e38b0518d99c21cece85f7402552a869c9362c86f7"
Nov 02 08:00:06.040599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:06.040577    4898 scope.go:122] "RemoveContainer" containerID="bfb64832b5dead644813d7e2dc111902aa63857520420485d6206d73af10ee44"
Nov 02 08:00:06.055187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:06.055160    4898 scope.go:122] "RemoveContainer" containerID="ca62f6d8f14d4f3bd22f5b8e02d33c727695ff85494193f9c49cb6c27f11cdf6"
Nov 02 08:00:06.072305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:06.072280    4898 scope.go:122] "RemoveContainer" containerID="5199cbfd064867fe4633e7cd00d1b86d5f103297b60572216700f7c52064c9df"
Nov 02 08:00:06.089137 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:06.089115    4898 scope.go:122] "RemoveContainer" containerID="51bb8374329b4fe641cdeaa1e8ed4c00a0938567502732d73e541c4aff757887"
Nov 02 08:00:06.104791 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:06.104688    4898 scope.go:122] "RemoveContainer" containerID="fdb8b9dd269e4944ff601b7dd45270ca91fead73a0535fc89fd70a72fec98cef"
Nov 02 08:00:07.007503 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:07.007467    4898 scope.go:122] "RemoveContainer" containerID="bd5f01825ddea9dd42858522d8574da79dbb20655c716d3f75e2255b2d809334"
Nov 02 08:00:07.008061 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:07.007858    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5_odh-ai-gateway-infra(42866b28-00e6-4781-a563-b42446b3e432)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5" podUID="42866b28-00e6-4781-a563-b42446b3e432"
Nov 02 08:00:07.014464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:07.014417    4898 scope.go:122] "RemoveContainer" containerID="6a3b4ba8e9ebd38cce57f93f55d3c123843bb54e4b9349c24b792248ea84d0eb"
Nov 02 08:00:07.014993 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:07.014943    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr_odh-ai-gateway-infra(92df2ddf-95bc-4fbd-ada9-df92a5ca62cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd"
Nov 02 08:00:07.017842 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:07.017815    4898 scope.go:122] "RemoveContainer" containerID="0a9205c5ce4dbdfbfc33b3d7c451951e26001a9c0d4b4eb5bf3e19fb4a89aaaa"
Nov 02 08:00:07.018220 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:07.018192    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4_odh-ai-gateway-infra(bbfacf71-eed8-4fde-96ef-c2e0c909fa30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30"
Nov 02 08:00:07.020200 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:07.020173    4898 scope.go:122] "RemoveContainer" containerID="6097a354ae1810e78ad4af1c38dec2c5cea23a4dfd423dea843c3c8b95ceb278"
Nov 02 08:00:07.020588 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:07.020559    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl_odh-ai-gateway-infra(93831e27-0aad-40de-babf-ce03c15daa13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl" podUID="93831e27-0aad-40de-babf-ce03c15daa13"
Nov 02 08:00:07.022741 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:07.022713    4898 scope.go:122] "RemoveContainer" containerID="99c588b6eb4b621de2616e3ad3d297e52fdfd063229db6f37fde266d7575a30f"
Nov 02 08:00:07.024026 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:07.023124    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx_odh-ai-gateway-infra(ce754bc4-4ab3-4b5b-8cb0-d35d080413bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd"
Nov 02 08:00:07.024870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:07.024850    4898 scope.go:122] "RemoveContainer" containerID="9975ffee1050a60e6e449c5e67c44fa112f03ef35b0e0582e947218286ffd9b6"
Nov 02 08:00:07.025123 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:07.025101    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc_odh-ai-gateway-infra(490dc0b1-9ca5-4f40-8639-e7480c7cce8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d"
Nov 02 08:00:07.027352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:07.027331    4898 scope.go:122] "RemoveContainer" containerID="b118ab2224d983ad7b15c61bf5387b9489a9e1f78d2ed30cabb413706bc9b631"
Nov 02 08:00:07.027600 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:07.027580    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t_odh-ai-gateway-infra(99f50886-1073-441c-ad74-1dcc66339adc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t" podUID="99f50886-1073-441c-ad74-1dcc66339adc"
Nov 02 08:00:07.045932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:07.042777    4898 scope.go:122] "RemoveContainer" containerID="599be23846f72271af112cb2a6d4f01ea5068f74d29bc87ad62cc6b949c2e37e"
Nov 02 08:00:07.045932 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:07.043122    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc_odh-ai-gateway-infra(5e59499e-9863-4849-bc47-e7a91d1fcf7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a"
Nov 02 08:00:07.045932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:07.045666    4898 scope.go:122] "RemoveContainer" containerID="a3f7929593ecebe830f4fa9c72e2675cd95c27590110f2243ee3a8d7bb9294ad"
Nov 02 08:00:07.046159 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:07.045964    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4_odh-ai-gateway-infra(ed459624-7446-4717-b618-ec235eb93ce7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4" podUID="ed459624-7446-4717-b618-ec235eb93ce7"
Nov 02 08:00:07.048951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:07.048195    4898 scope.go:122] "RemoveContainer" containerID="ffc7ead0678f2958118d08d417591f833139db1d153cb82336a0d00b433363b9"
Nov 02 08:00:07.048951 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:07.048435    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893440-28szk_odh-ai-gateway-infra(516fa8a4-8039-4ca0-ac0c-83817acc91c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3"
Nov 02 08:00:07.050721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:07.050663    4898 scope.go:122] "RemoveContainer" containerID="81bd1faf86f5ae1ee2a8be7e4b03cc44197958469ae3a030885cb83acd5c3d5f"
Nov 02 08:00:07.050958 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:07.050932    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545_odh-ai-gateway-infra(f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb"
Nov 02 08:00:10.109126 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:10.109097    4898 generic.go:356] "Generic (PLEG): container finished" podID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerID="8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456" exitCode=0
Nov 02 08:00:10.109520 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:10.109170    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" event={"ID":"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09","Type":"ContainerDied","Data":"8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456"}
Nov 02 08:00:10.109684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:10.109665    4898 scope.go:122] "RemoveContainer" containerID="8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456"
Nov 02 08:00:10.111309 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:10.111281    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:10.111369 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:10.111354    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:10.111420 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:10.111410    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:11.121686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:11.121646    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" event={"ID":"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09","Type":"ContainerStarted","Data":"21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd"}
Nov 02 08:00:11.189032 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:11.188976    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 08:00:11.192018 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:11.191984    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 02 08:00:11.192280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:11.192037    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 02 08:00:12.189955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:12.187274    4898 scope.go:122] "RemoveContainer" containerID="c6fe8ec035b9c9fa85c012cbe761c7b7984d5b50530f07078c1713f9eee8b6bd"
Nov 02 08:00:12.190513 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:12.190010    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 02 08:00:12.190513 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:12.190063    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 02 08:00:12.204170 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:12.204140    4898 scope.go:122] "RemoveContainer" containerID="f1c16e6d0869ead83779fe5ca9e734ed688660ea9ec84add3a559d09bfbc1e07"
Nov 02 08:00:12.840870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:12.840823    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 08:00:13.202909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:13.193276    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 02 08:00:13.202909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:13.193341    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 02 08:00:14.187423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:14.187386    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 02 08:00:14.187620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:14.187442    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 02 08:00:15.011825 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:15.011783    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:00:15.190149 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:15.190097    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 02 08:00:15.190448 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:15.190408    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 02 08:00:15.202569 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:15.202539    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"]
Nov 02 08:00:15.487947 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:15.487917    4898 scope.go:122] "RemoveContainer" containerID="599be23846f72271af112cb2a6d4f01ea5068f74d29bc87ad62cc6b949c2e37e"
Nov 02 08:00:16.187637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:16.187589    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Nov 02 08:00:16.188092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:16.187660    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Nov 02 08:00:16.202485 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:16.202451    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc" event={"ID":"5e59499e-9863-4849-bc47-e7a91d1fcf7a","Type":"ContainerStarted","Data":"5fada982561518ff7f4fa01f59d281c87738073c7ced477dda2420c6c45ff5f8"}
Nov 02 08:00:16.458839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:16.457422    4898 scope.go:122] "RemoveContainer" containerID="bd5f01825ddea9dd42858522d8574da79dbb20655c716d3f75e2255b2d809334"
Nov 02 08:00:16.459279 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:16.459247    4898 scope.go:122] "RemoveContainer" containerID="99c588b6eb4b621de2616e3ad3d297e52fdfd063229db6f37fde266d7575a30f"
Nov 02 08:00:16.460063 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:16.459713    4898 scope.go:122] "RemoveContainer" containerID="b118ab2224d983ad7b15c61bf5387b9489a9e1f78d2ed30cabb413706bc9b631"
Nov 02 08:00:17.191557 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:17.191525    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 08:00:17.192927 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:17.192902    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 08:00:17.213743 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:17.213706    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5" event={"ID":"42866b28-00e6-4781-a563-b42446b3e432","Type":"ContainerStarted","Data":"716b8a9ed4e754cd81e469bf221b618394f9325f9fbfb91d8e332a5165fe7f42"}
Nov 02 08:00:17.215898 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:17.215863    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx" event={"ID":"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd","Type":"ContainerStarted","Data":"c151d20afac04883ef3846983e97e3819e01679bce9058cd1b296ced92e01566"}
Nov 02 08:00:17.218142 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:17.218116    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t" event={"ID":"99f50886-1073-441c-ad74-1dcc66339adc","Type":"ContainerStarted","Data":"2d0a05bf2caaf71f3d29b17ab198fde1a2e3cdbbb9c2e7a2bea68186a6609744"}
Nov 02 08:00:17.220203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:17.220180    4898 generic.go:356] "Generic (PLEG): container finished" podID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" containerID="5fada982561518ff7f4fa01f59d281c87738073c7ced477dda2420c6c45ff5f8" exitCode=60
Nov 02 08:00:17.220303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:17.220251    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc" event={"ID":"5e59499e-9863-4849-bc47-e7a91d1fcf7a","Type":"ContainerDied","Data":"5fada982561518ff7f4fa01f59d281c87738073c7ced477dda2420c6c45ff5f8"}
Nov 02 08:00:17.220373 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:17.220302    4898 scope.go:122] "RemoveContainer" containerID="599be23846f72271af112cb2a6d4f01ea5068f74d29bc87ad62cc6b949c2e37e"
Nov 02 08:00:17.221156 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:17.221135    4898 scope.go:122] "RemoveContainer" containerID="5fada982561518ff7f4fa01f59d281c87738073c7ced477dda2420c6c45ff5f8"
Nov 02 08:00:17.221458 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:17.221436    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc_odh-ai-gateway-infra(5e59499e-9863-4849-bc47-e7a91d1fcf7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a"
Nov 02 08:00:17.269874 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:17.269840    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"]
Nov 02 08:00:17.587908 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:17.587830    4898 scope.go:122] "RemoveContainer" containerID="81bd1faf86f5ae1ee2a8be7e4b03cc44197958469ae3a030885cb83acd5c3d5f"
Nov 02 08:00:17.744638 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:17.744562    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce754bc4_4ab3_4b5b_8cb0_d35d080413bd.slice/crio-conmon-c151d20afac04883ef3846983e97e3819e01679bce9058cd1b296ced92e01566.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 08:00:18.230344 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.230312    4898 generic.go:356] "Generic (PLEG): container finished" podID="99f50886-1073-441c-ad74-1dcc66339adc" containerID="2d0a05bf2caaf71f3d29b17ab198fde1a2e3cdbbb9c2e7a2bea68186a6609744" exitCode=60
Nov 02 08:00:18.230839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.230385    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t" event={"ID":"99f50886-1073-441c-ad74-1dcc66339adc","Type":"ContainerDied","Data":"2d0a05bf2caaf71f3d29b17ab198fde1a2e3cdbbb9c2e7a2bea68186a6609744"}
Nov 02 08:00:18.230839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.230458    4898 scope.go:122] "RemoveContainer" containerID="b118ab2224d983ad7b15c61bf5387b9489a9e1f78d2ed30cabb413706bc9b631"
Nov 02 08:00:18.230997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.230915    4898 scope.go:122] "RemoveContainer" containerID="2d0a05bf2caaf71f3d29b17ab198fde1a2e3cdbbb9c2e7a2bea68186a6609744"
Nov 02 08:00:18.231250 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:18.231228    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t_odh-ai-gateway-infra(99f50886-1073-441c-ad74-1dcc66339adc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t" podUID="99f50886-1073-441c-ad74-1dcc66339adc"
Nov 02 08:00:18.234818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.234789    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545" event={"ID":"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb","Type":"ContainerStarted","Data":"cf7b2da43c8dd55e54295cc21d2ee88dd24b02dad369af60635c6f40ee84e6ed"}
Nov 02 08:00:18.237047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.237027    4898 generic.go:356] "Generic (PLEG): container finished" podID="42866b28-00e6-4781-a563-b42446b3e432" containerID="716b8a9ed4e754cd81e469bf221b618394f9325f9fbfb91d8e332a5165fe7f42" exitCode=60
Nov 02 08:00:18.237447 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.237097    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5" event={"ID":"42866b28-00e6-4781-a563-b42446b3e432","Type":"ContainerDied","Data":"716b8a9ed4e754cd81e469bf221b618394f9325f9fbfb91d8e332a5165fe7f42"}
Nov 02 08:00:18.237447 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.237411    4898 scope.go:122] "RemoveContainer" containerID="716b8a9ed4e754cd81e469bf221b618394f9325f9fbfb91d8e332a5165fe7f42"
Nov 02 08:00:18.237682 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:18.237660    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5_odh-ai-gateway-infra(42866b28-00e6-4781-a563-b42446b3e432)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5" podUID="42866b28-00e6-4781-a563-b42446b3e432"
Nov 02 08:00:18.239949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.239871    4898 generic.go:356] "Generic (PLEG): container finished" podID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" containerID="c151d20afac04883ef3846983e97e3819e01679bce9058cd1b296ced92e01566" exitCode=60
Nov 02 08:00:18.239949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.239911    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx" event={"ID":"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd","Type":"ContainerDied","Data":"c151d20afac04883ef3846983e97e3819e01679bce9058cd1b296ced92e01566"}
Nov 02 08:00:18.240214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.240198    4898 scope.go:122] "RemoveContainer" containerID="c151d20afac04883ef3846983e97e3819e01679bce9058cd1b296ced92e01566"
Nov 02 08:00:18.240439 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:18.240418    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx_odh-ai-gateway-infra(ce754bc4-4ab3-4b5b-8cb0-d35d080413bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd"
Nov 02 08:00:18.245581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.245557    4898 scope.go:122] "RemoveContainer" containerID="bd5f01825ddea9dd42858522d8574da79dbb20655c716d3f75e2255b2d809334"
Nov 02 08:00:18.262455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.262432    4898 scope.go:122] "RemoveContainer" containerID="99c588b6eb4b621de2616e3ad3d297e52fdfd063229db6f37fde266d7575a30f"
Nov 02 08:00:18.363030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.362997    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"]
Nov 02 08:00:18.457785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.457710    4898 scope.go:122] "RemoveContainer" containerID="6a3b4ba8e9ebd38cce57f93f55d3c123843bb54e4b9349c24b792248ea84d0eb"
Nov 02 08:00:18.458148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.458126    4898 scope.go:122] "RemoveContainer" containerID="6097a354ae1810e78ad4af1c38dec2c5cea23a4dfd423dea843c3c8b95ceb278"
Nov 02 08:00:18.459115 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.459093    4898 scope.go:122] "RemoveContainer" containerID="a3f7929593ecebe830f4fa9c72e2675cd95c27590110f2243ee3a8d7bb9294ad"
Nov 02 08:00:18.537472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.537443    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"
Nov 02 08:00:18.558614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.558583    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"]
Nov 02 08:00:18.600298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.600258    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e59499e-9863-4849-bc47-e7a91d1fcf7a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e59499e-9863-4849-bc47-e7a91d1fcf7a-maas-api-service-ca\") pod \"5e59499e-9863-4849-bc47-e7a91d1fcf7a\" (UID: \"5e59499e-9863-4849-bc47-e7a91d1fcf7a\") "
Nov 02 08:00:18.600502 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.600393    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e59499e-9863-4849-bc47-e7a91d1fcf7a-kube-api-access-f7grk\" (UniqueName: \"kubernetes.io/projected/5e59499e-9863-4849-bc47-e7a91d1fcf7a-kube-api-access-f7grk\") pod \"5e59499e-9863-4849-bc47-e7a91d1fcf7a\" (UID: \"5e59499e-9863-4849-bc47-e7a91d1fcf7a\") "
Nov 02 08:00:18.600710 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.600684    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e59499e-9863-4849-bc47-e7a91d1fcf7a-maas-api-service-ca" pod "5e59499e-9863-4849-bc47-e7a91d1fcf7a" (UID: "5e59499e-9863-4849-bc47-e7a91d1fcf7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:18.601198 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.601148    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e59499e-9863-4849-bc47-e7a91d1fcf7a-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:18.615225 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.615196    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"]
Nov 02 08:00:18.664199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.664150    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e59499e-9863-4849-bc47-e7a91d1fcf7a-kube-api-access-f7grk" pod "5e59499e-9863-4849-bc47-e7a91d1fcf7a" (UID: "5e59499e-9863-4849-bc47-e7a91d1fcf7a"). InnerVolumeSpecName "kube-api-access-f7grk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:18.706185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:18.706153    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7grk\" (UniqueName: \"kubernetes.io/projected/5e59499e-9863-4849-bc47-e7a91d1fcf7a-kube-api-access-f7grk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:19.252984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.252959    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"
Nov 02 08:00:19.253401 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.252981    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc" event={"ID":"5e59499e-9863-4849-bc47-e7a91d1fcf7a","Type":"ContainerDied","Data":"423b403f0dbccaf05f578310ed6495f363d6e16a06e6917f6ea8f789bf362472"}
Nov 02 08:00:19.253401 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.253045    4898 scope.go:122] "RemoveContainer" containerID="5fada982561518ff7f4fa01f59d281c87738073c7ced477dda2420c6c45ff5f8"
Nov 02 08:00:19.260180 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.260139    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4" event={"ID":"ed459624-7446-4717-b618-ec235eb93ce7","Type":"ContainerStarted","Data":"c3dc24545d0ab279a2ab8dd1524cd690083f9cd2d191e5caa60da25125524575"}
Nov 02 08:00:19.263444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.263415    4898 generic.go:356] "Generic (PLEG): container finished" podID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" containerID="cf7b2da43c8dd55e54295cc21d2ee88dd24b02dad369af60635c6f40ee84e6ed" exitCode=60
Nov 02 08:00:19.263563 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.263512    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545" event={"ID":"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb","Type":"ContainerDied","Data":"cf7b2da43c8dd55e54295cc21d2ee88dd24b02dad369af60635c6f40ee84e6ed"}
Nov 02 08:00:19.263956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.263934    4898 scope.go:122] "RemoveContainer" containerID="cf7b2da43c8dd55e54295cc21d2ee88dd24b02dad369af60635c6f40ee84e6ed"
Nov 02 08:00:19.264464 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:19.264433    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545_odh-ai-gateway-infra(f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb"
Nov 02 08:00:19.268911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.268860    4898 scope.go:122] "RemoveContainer" containerID="81bd1faf86f5ae1ee2a8be7e4b03cc44197958469ae3a030885cb83acd5c3d5f"
Nov 02 08:00:19.271608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.271581    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr" event={"ID":"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd","Type":"ContainerStarted","Data":"6ccac0c86144a6888e75b0906542d91b3dbfffaf1f843b5805c541d0a9007bce"}
Nov 02 08:00:19.277108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.276322    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl" event={"ID":"93831e27-0aad-40de-babf-ce03c15daa13","Type":"ContainerStarted","Data":"b40e2dbff54cfb09dae97aad6764960e2c0b87a1d43af89482703d31ece65987"}
Nov 02 08:00:19.401122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.401080    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"]
Nov 02 08:00:19.407553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.407523    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893440-db5tc"]
Nov 02 08:00:19.459504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.457126    4898 scope.go:122] "RemoveContainer" containerID="9975ffee1050a60e6e449c5e67c44fa112f03ef35b0e0582e947218286ffd9b6"
Nov 02 08:00:19.459504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.457616    4898 scope.go:122] "RemoveContainer" containerID="ffc7ead0678f2958118d08d417591f833139db1d153cb82336a0d00b433363b9"
Nov 02 08:00:19.459504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.458080    4898 scope.go:122] "RemoveContainer" containerID="0a9205c5ce4dbdfbfc33b3d7c451951e26001a9c0d4b4eb5bf3e19fb4a89aaaa"
Nov 02 08:00:19.467681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.464215    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"
Nov 02 08:00:19.474874 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.474018    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" path="/var/lib/kubelet/pods/5e59499e-9863-4849-bc47-e7a91d1fcf7a/volumes"
Nov 02 08:00:19.474874 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.474448    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"]
Nov 02 08:00:19.524962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.523841    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-maas-api-service-ca\") pod \"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd\" (UID: \"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd\") "
Nov 02 08:00:19.524962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.523930    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-kube-api-access-9r6gz\" (UniqueName: \"kubernetes.io/projected/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-kube-api-access-9r6gz\") pod \"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd\" (UID: \"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd\") "
Nov 02 08:00:19.525912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.525292    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-maas-api-service-ca" pod "ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" (UID: "ce754bc4-4ab3-4b5b-8cb0-d35d080413bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:19.530186 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.530160    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-kube-api-access-9r6gz" pod "ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" (UID: "ce754bc4-4ab3-4b5b-8cb0-d35d080413bd"). InnerVolumeSpecName "kube-api-access-9r6gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:19.558190 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.555546    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"
Nov 02 08:00:19.558190 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.558067    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"
Nov 02 08:00:19.625628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.625444    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f50886-1073-441c-ad74-1dcc66339adc-kube-api-access-fwbjx\" (UniqueName: \"kubernetes.io/projected/99f50886-1073-441c-ad74-1dcc66339adc-kube-api-access-fwbjx\") pod \"99f50886-1073-441c-ad74-1dcc66339adc\" (UID: \"99f50886-1073-441c-ad74-1dcc66339adc\") "
Nov 02 08:00:19.625628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.625509    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42866b28-00e6-4781-a563-b42446b3e432-kube-api-access-mgh5z\" (UniqueName: \"kubernetes.io/projected/42866b28-00e6-4781-a563-b42446b3e432-kube-api-access-mgh5z\") pod \"42866b28-00e6-4781-a563-b42446b3e432\" (UID: \"42866b28-00e6-4781-a563-b42446b3e432\") "
Nov 02 08:00:19.626116 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.625706    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f50886-1073-441c-ad74-1dcc66339adc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99f50886-1073-441c-ad74-1dcc66339adc-maas-api-service-ca\") pod \"99f50886-1073-441c-ad74-1dcc66339adc\" (UID: \"99f50886-1073-441c-ad74-1dcc66339adc\") "
Nov 02 08:00:19.626116 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.625768    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42866b28-00e6-4781-a563-b42446b3e432-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42866b28-00e6-4781-a563-b42446b3e432-maas-api-service-ca\") pod \"42866b28-00e6-4781-a563-b42446b3e432\" (UID: \"42866b28-00e6-4781-a563-b42446b3e432\") "
Nov 02 08:00:19.626244 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.626147    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f50886-1073-441c-ad74-1dcc66339adc-maas-api-service-ca" pod "99f50886-1073-441c-ad74-1dcc66339adc" (UID: "99f50886-1073-441c-ad74-1dcc66339adc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:19.626316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.626272    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42866b28-00e6-4781-a563-b42446b3e432-maas-api-service-ca" pod "42866b28-00e6-4781-a563-b42446b3e432" (UID: "42866b28-00e6-4781-a563-b42446b3e432"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:19.626480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.626460    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99f50886-1073-441c-ad74-1dcc66339adc-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:19.626563 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.626486    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42866b28-00e6-4781-a563-b42446b3e432-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:19.626563 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.626504    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:19.626563 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.626519    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r6gz\" (UniqueName: \"kubernetes.io/projected/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd-kube-api-access-9r6gz\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:19.628218 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.628197    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f50886-1073-441c-ad74-1dcc66339adc-kube-api-access-fwbjx" pod "99f50886-1073-441c-ad74-1dcc66339adc" (UID: "99f50886-1073-441c-ad74-1dcc66339adc"). InnerVolumeSpecName "kube-api-access-fwbjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:19.628361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.628346    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42866b28-00e6-4781-a563-b42446b3e432-kube-api-access-mgh5z" pod "42866b28-00e6-4781-a563-b42446b3e432" (UID: "42866b28-00e6-4781-a563-b42446b3e432"). InnerVolumeSpecName "kube-api-access-mgh5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:19.728810 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.728770    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwbjx\" (UniqueName: \"kubernetes.io/projected/99f50886-1073-441c-ad74-1dcc66339adc-kube-api-access-fwbjx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:19.728810 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:19.728810    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgh5z\" (UniqueName: \"kubernetes.io/projected/42866b28-00e6-4781-a563-b42446b3e432-kube-api-access-mgh5z\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:20.286144 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.286113    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"
Nov 02 08:00:20.286613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.286140    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t" event={"ID":"99f50886-1073-441c-ad74-1dcc66339adc","Type":"ContainerDied","Data":"a5204dc1a94d30311f537236f9365cbf5fd47421b89cfb05d83570692fafcbf1"}
Nov 02 08:00:20.286613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.286216    4898 scope.go:122] "RemoveContainer" containerID="2d0a05bf2caaf71f3d29b17ab198fde1a2e3cdbbb9c2e7a2bea68186a6609744"
Nov 02 08:00:20.289604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.289579    4898 generic.go:356] "Generic (PLEG): container finished" podID="ed459624-7446-4717-b618-ec235eb93ce7" containerID="c3dc24545d0ab279a2ab8dd1524cd690083f9cd2d191e5caa60da25125524575" exitCode=60
Nov 02 08:00:20.289742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.289697    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4" event={"ID":"ed459624-7446-4717-b618-ec235eb93ce7","Type":"ContainerDied","Data":"c3dc24545d0ab279a2ab8dd1524cd690083f9cd2d191e5caa60da25125524575"}
Nov 02 08:00:20.290094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.290072    4898 scope.go:122] "RemoveContainer" containerID="c3dc24545d0ab279a2ab8dd1524cd690083f9cd2d191e5caa60da25125524575"
Nov 02 08:00:20.290445 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:20.290416    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4_odh-ai-gateway-infra(ed459624-7446-4717-b618-ec235eb93ce7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4" podUID="ed459624-7446-4717-b618-ec235eb93ce7"
Nov 02 08:00:20.292717 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.292677    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk" event={"ID":"516fa8a4-8039-4ca0-ac0c-83817acc91c3","Type":"ContainerStarted","Data":"5056eb36ed95d930c1da24f19660f1465ceba47fcdec6ae51ea04ebd4638d6d9"}
Nov 02 08:00:20.296299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.296272    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"
Nov 02 08:00:20.296411 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.296295    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5" event={"ID":"42866b28-00e6-4781-a563-b42446b3e432","Type":"ContainerDied","Data":"43b453274512719cfa0e0d8eff56e6135fc632ccc8c8b985102826e50b503159"}
Nov 02 08:00:20.298880 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.298858    4898 generic.go:356] "Generic (PLEG): container finished" podID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" containerID="6ccac0c86144a6888e75b0906542d91b3dbfffaf1f843b5805c541d0a9007bce" exitCode=60
Nov 02 08:00:20.299045 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.298933    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr" event={"ID":"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd","Type":"ContainerDied","Data":"6ccac0c86144a6888e75b0906542d91b3dbfffaf1f843b5805c541d0a9007bce"}
Nov 02 08:00:20.299720 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.299701    4898 scope.go:122] "RemoveContainer" containerID="6ccac0c86144a6888e75b0906542d91b3dbfffaf1f843b5805c541d0a9007bce"
Nov 02 08:00:20.299985 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:20.299965    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr_odh-ai-gateway-infra(92df2ddf-95bc-4fbd-ada9-df92a5ca62cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd"
Nov 02 08:00:20.300427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.300409    4898 scope.go:122] "RemoveContainer" containerID="a3f7929593ecebe830f4fa9c72e2675cd95c27590110f2243ee3a8d7bb9294ad"
Nov 02 08:00:20.301830 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.301807    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4" event={"ID":"bbfacf71-eed8-4fde-96ef-c2e0c909fa30","Type":"ContainerStarted","Data":"aa597c37acd24f185cadb5d93239b1b1824facdd18f59505211db6364565159d"}
Nov 02 08:00:20.304260 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.304237    4898 generic.go:356] "Generic (PLEG): container finished" podID="93831e27-0aad-40de-babf-ce03c15daa13" containerID="b40e2dbff54cfb09dae97aad6764960e2c0b87a1d43af89482703d31ece65987" exitCode=60
Nov 02 08:00:20.304371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.304311    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl" event={"ID":"93831e27-0aad-40de-babf-ce03c15daa13","Type":"ContainerDied","Data":"b40e2dbff54cfb09dae97aad6764960e2c0b87a1d43af89482703d31ece65987"}
Nov 02 08:00:20.309919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.304638    4898 scope.go:122] "RemoveContainer" containerID="b40e2dbff54cfb09dae97aad6764960e2c0b87a1d43af89482703d31ece65987"
Nov 02 08:00:20.309919 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:20.304984    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl_odh-ai-gateway-infra(93831e27-0aad-40de-babf-ce03c15daa13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl" podUID="93831e27-0aad-40de-babf-ce03c15daa13"
Nov 02 08:00:20.309919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.306049    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"
Nov 02 08:00:20.309919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.306105    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx" event={"ID":"ce754bc4-4ab3-4b5b-8cb0-d35d080413bd","Type":"ContainerDied","Data":"c934da84132fb1f2cd826e32df0988f42154bc605f2fce06bae56980d0537419"}
Nov 02 08:00:20.316656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.316620    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc" event={"ID":"490dc0b1-9ca5-4f40-8639-e7480c7cce8d","Type":"ContainerStarted","Data":"c83136e26087d70488f63944a2ceb8bb28cc0f34810ae9b2bd55c64fe7670eae"}
Nov 02 08:00:20.318033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.318014    4898 scope.go:122] "RemoveContainer" containerID="716b8a9ed4e754cd81e469bf221b618394f9325f9fbfb91d8e332a5165fe7f42"
Nov 02 08:00:20.334145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.334023    4898 scope.go:122] "RemoveContainer" containerID="6a3b4ba8e9ebd38cce57f93f55d3c123843bb54e4b9349c24b792248ea84d0eb"
Nov 02 08:00:20.342441 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.342412    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"]
Nov 02 08:00:20.347262 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.347234    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893440-svw9t"]
Nov 02 08:00:20.360746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.360686    4898 scope.go:122] "RemoveContainer" containerID="6097a354ae1810e78ad4af1c38dec2c5cea23a4dfd423dea843c3c8b95ceb278"
Nov 02 08:00:20.388792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.385545    4898 scope.go:122] "RemoveContainer" containerID="c151d20afac04883ef3846983e97e3819e01679bce9058cd1b296ced92e01566"
Nov 02 08:00:20.452652 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.452574    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"]
Nov 02 08:00:20.525584 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.525554    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"
Nov 02 08:00:20.562572 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.562544    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"]
Nov 02 08:00:20.640202 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.640168    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"]
Nov 02 08:00:20.648690 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.648664    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"]
Nov 02 08:00:20.650688 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.650650    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-kube-api-access-r8xh2\" (UniqueName: \"kubernetes.io/projected/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-kube-api-access-r8xh2\") pod \"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb\" (UID: \"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb\") "
Nov 02 08:00:20.650962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.650936    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-maas-api-service-ca\") pod \"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb\" (UID: \"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb\") "
Nov 02 08:00:20.651368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.651343    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-maas-api-service-ca" pod "f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" (UID: "f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:20.651534 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.651513    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:20.653446 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.653424    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893440-24wr5"]
Nov 02 08:00:20.653593 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.653570    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-kube-api-access-r8xh2" pod "f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" (UID: "f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb"). InnerVolumeSpecName "kube-api-access-r8xh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:20.690589 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.690558    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"]
Nov 02 08:00:20.698657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.698631    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893440-bqbdx"]
Nov 02 08:00:20.753282 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:20.753202    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8xh2\" (UniqueName: \"kubernetes.io/projected/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb-kube-api-access-r8xh2\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:21.333145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.333091    4898 generic.go:356] "Generic (PLEG): container finished" podID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" containerID="aa597c37acd24f185cadb5d93239b1b1824facdd18f59505211db6364565159d" exitCode=60
Nov 02 08:00:21.336648 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.336607    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4" event={"ID":"bbfacf71-eed8-4fde-96ef-c2e0c909fa30","Type":"ContainerDied","Data":"aa597c37acd24f185cadb5d93239b1b1824facdd18f59505211db6364565159d"}
Nov 02 08:00:21.337060 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.337029    4898 scope.go:122] "RemoveContainer" containerID="0a9205c5ce4dbdfbfc33b3d7c451951e26001a9c0d4b4eb5bf3e19fb4a89aaaa"
Nov 02 08:00:21.337730 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.337711    4898 scope.go:122] "RemoveContainer" containerID="aa597c37acd24f185cadb5d93239b1b1824facdd18f59505211db6364565159d"
Nov 02 08:00:21.338286 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:21.338260    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4_odh-ai-gateway-infra(bbfacf71-eed8-4fde-96ef-c2e0c909fa30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30"
Nov 02 08:00:21.371648 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.371618    4898 generic.go:356] "Generic (PLEG): container finished" podID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" containerID="c83136e26087d70488f63944a2ceb8bb28cc0f34810ae9b2bd55c64fe7670eae" exitCode=60
Nov 02 08:00:21.371790 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.371690    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc" event={"ID":"490dc0b1-9ca5-4f40-8639-e7480c7cce8d","Type":"ContainerDied","Data":"c83136e26087d70488f63944a2ceb8bb28cc0f34810ae9b2bd55c64fe7670eae"}
Nov 02 08:00:21.371790 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.371748    4898 scope.go:122] "RemoveContainer" containerID="9975ffee1050a60e6e449c5e67c44fa112f03ef35b0e0582e947218286ffd9b6"
Nov 02 08:00:21.372218 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.372199    4898 scope.go:122] "RemoveContainer" containerID="c83136e26087d70488f63944a2ceb8bb28cc0f34810ae9b2bd55c64fe7670eae"
Nov 02 08:00:21.372554 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:21.372528    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc_odh-ai-gateway-infra(490dc0b1-9ca5-4f40-8639-e7480c7cce8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d"
Nov 02 08:00:21.378225 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.378135    4898 generic.go:356] "Generic (PLEG): container finished" podID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" containerID="5056eb36ed95d930c1da24f19660f1465ceba47fcdec6ae51ea04ebd4638d6d9" exitCode=60
Nov 02 08:00:21.378485 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.378430    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk" event={"ID":"516fa8a4-8039-4ca0-ac0c-83817acc91c3","Type":"ContainerDied","Data":"5056eb36ed95d930c1da24f19660f1465ceba47fcdec6ae51ea04ebd4638d6d9"}
Nov 02 08:00:21.379238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.379200    4898 scope.go:122] "RemoveContainer" containerID="5056eb36ed95d930c1da24f19660f1465ceba47fcdec6ae51ea04ebd4638d6d9"
Nov 02 08:00:21.379638 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:21.379612    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893440-28szk_odh-ai-gateway-infra(516fa8a4-8039-4ca0-ac0c-83817acc91c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3"
Nov 02 08:00:21.381402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.381373    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545" event={"ID":"f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb","Type":"ContainerDied","Data":"899aac9cc80172fafd601ad0f063ade63099161876ee75e388249461dc7fcea1"}
Nov 02 08:00:21.381510 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.381475    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"
Nov 02 08:00:21.395436 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.395412    4898 scope.go:122] "RemoveContainer" containerID="ffc7ead0678f2958118d08d417591f833139db1d153cb82336a0d00b433363b9"
Nov 02 08:00:21.421074 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.420937    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"]
Nov 02 08:00:21.429901 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.428997    4898 scope.go:122] "RemoveContainer" containerID="cf7b2da43c8dd55e54295cc21d2ee88dd24b02dad369af60635c6f40ee84e6ed"
Nov 02 08:00:21.490364 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.489824    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42866b28-00e6-4781-a563-b42446b3e432" path="/var/lib/kubelet/pods/42866b28-00e6-4781-a563-b42446b3e432/volumes"
Nov 02 08:00:21.491634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.491035    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f50886-1073-441c-ad74-1dcc66339adc" path="/var/lib/kubelet/pods/99f50886-1073-441c-ad74-1dcc66339adc/volumes"
Nov 02 08:00:21.491634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.491610    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" path="/var/lib/kubelet/pods/ce754bc4-4ab3-4b5b-8cb0-d35d080413bd/volumes"
Nov 02 08:00:21.581883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.581856    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"
Nov 02 08:00:21.669671 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.669641    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93831e27-0aad-40de-babf-ce03c15daa13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93831e27-0aad-40de-babf-ce03c15daa13-maas-api-service-ca\") pod \"93831e27-0aad-40de-babf-ce03c15daa13\" (UID: \"93831e27-0aad-40de-babf-ce03c15daa13\") "
Nov 02 08:00:21.669829 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.669747    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93831e27-0aad-40de-babf-ce03c15daa13-kube-api-access-hmhn8\" (UniqueName: \"kubernetes.io/projected/93831e27-0aad-40de-babf-ce03c15daa13-kube-api-access-hmhn8\") pod \"93831e27-0aad-40de-babf-ce03c15daa13\" (UID: \"93831e27-0aad-40de-babf-ce03c15daa13\") "
Nov 02 08:00:21.670990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.670964    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93831e27-0aad-40de-babf-ce03c15daa13-maas-api-service-ca" pod "93831e27-0aad-40de-babf-ce03c15daa13" (UID: "93831e27-0aad-40de-babf-ce03c15daa13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:21.674179 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.674144    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93831e27-0aad-40de-babf-ce03c15daa13-kube-api-access-hmhn8" pod "93831e27-0aad-40de-babf-ce03c15daa13" (UID: "93831e27-0aad-40de-babf-ce03c15daa13"). InnerVolumeSpecName "kube-api-access-hmhn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:21.680174 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.680148    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"
Nov 02 08:00:21.690180 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.690151    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"
Nov 02 08:00:21.692877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.692854    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"]
Nov 02 08:00:21.706766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.706741    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-56bbb6-29893440-rn545"]
Nov 02 08:00:21.770730 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.770696    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed459624-7446-4717-b618-ec235eb93ce7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed459624-7446-4717-b618-ec235eb93ce7-maas-api-service-ca\") pod \"ed459624-7446-4717-b618-ec235eb93ce7\" (UID: \"ed459624-7446-4717-b618-ec235eb93ce7\") "
Nov 02 08:00:21.770925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.770833    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-kube-api-access-4fwhx\" (UniqueName: \"kubernetes.io/projected/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-kube-api-access-4fwhx\") pod \"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd\" (UID: \"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd\") "
Nov 02 08:00:21.770925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.770908    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-maas-api-service-ca\") pod \"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd\" (UID: \"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd\") "
Nov 02 08:00:21.771051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.770927    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed459624-7446-4717-b618-ec235eb93ce7-kube-api-access-dnwjj\" (UniqueName: \"kubernetes.io/projected/ed459624-7446-4717-b618-ec235eb93ce7-kube-api-access-dnwjj\") pod \"ed459624-7446-4717-b618-ec235eb93ce7\" (UID: \"ed459624-7446-4717-b618-ec235eb93ce7\") "
Nov 02 08:00:21.771402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.771363    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmhn8\" (UniqueName: \"kubernetes.io/projected/93831e27-0aad-40de-babf-ce03c15daa13-kube-api-access-hmhn8\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:21.771402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.771391    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93831e27-0aad-40de-babf-ce03c15daa13-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:21.771615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.771500    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed459624-7446-4717-b618-ec235eb93ce7-maas-api-service-ca" pod "ed459624-7446-4717-b618-ec235eb93ce7" (UID: "ed459624-7446-4717-b618-ec235eb93ce7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:21.771615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.771537    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-maas-api-service-ca" pod "92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" (UID: "92df2ddf-95bc-4fbd-ada9-df92a5ca62cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:21.773384 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.773362    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed459624-7446-4717-b618-ec235eb93ce7-kube-api-access-dnwjj" pod "ed459624-7446-4717-b618-ec235eb93ce7" (UID: "ed459624-7446-4717-b618-ec235eb93ce7"). InnerVolumeSpecName "kube-api-access-dnwjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:21.773520 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.773506    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-kube-api-access-4fwhx" pod "92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" (UID: "92df2ddf-95bc-4fbd-ada9-df92a5ca62cd"). InnerVolumeSpecName "kube-api-access-4fwhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:21.849420 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.849385    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"]
Nov 02 08:00:21.873137 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.873101    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed459624-7446-4717-b618-ec235eb93ce7-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:21.873137 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.873132    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fwhx\" (UniqueName: \"kubernetes.io/projected/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-kube-api-access-4fwhx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:21.873137 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.873144    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:21.873423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.873155    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnwjj\" (UniqueName: \"kubernetes.io/projected/ed459624-7446-4717-b618-ec235eb93ce7-kube-api-access-dnwjj\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:21.873423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:21.873188    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"]
Nov 02 08:00:22.375973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.375839    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"]
Nov 02 08:00:22.401363 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.401327    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" containerName="cleanup"
Nov 02 08:00:22.401578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.401561    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" containerName="cleanup"
Nov 02 08:00:22.401694 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.401684    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" containerName="cleanup"
Nov 02 08:00:22.401774 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.401765    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" containerName="cleanup"
Nov 02 08:00:22.401857 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.401844    4898 cpu_manager.go:414] "containerMap: removing container" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" containerName="cleanup"
Nov 02 08:00:22.401968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.401947    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" containerName="cleanup"
Nov 02 08:00:22.402053 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.402044    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ed459624-7446-4717-b618-ec235eb93ce7" containerName="cleanup"
Nov 02 08:00:22.402131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.402121    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed459624-7446-4717-b618-ec235eb93ce7" containerName="cleanup"
Nov 02 08:00:22.402212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.402203    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" containerName="cleanup"
Nov 02 08:00:22.402285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.402276    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" containerName="cleanup"
Nov 02 08:00:22.402378 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.402368    4898 cpu_manager.go:414] "containerMap: removing container" podUID="93831e27-0aad-40de-babf-ce03c15daa13" containerName="cleanup"
Nov 02 08:00:22.402457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.402447    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93831e27-0aad-40de-babf-ce03c15daa13" containerName="cleanup"
Nov 02 08:00:22.402536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.402527    4898 cpu_manager.go:414] "containerMap: removing container" podUID="99f50886-1073-441c-ad74-1dcc66339adc" containerName="cleanup"
Nov 02 08:00:22.402609 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.402600    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f50886-1073-441c-ad74-1dcc66339adc" containerName="cleanup"
Nov 02 08:00:22.402691 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.402682    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ed459624-7446-4717-b618-ec235eb93ce7" containerName="cleanup"
Nov 02 08:00:22.402764 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.402755    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed459624-7446-4717-b618-ec235eb93ce7" containerName="cleanup"
Nov 02 08:00:22.402848 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.402839    4898 cpu_manager.go:414] "containerMap: removing container" podUID="93831e27-0aad-40de-babf-ce03c15daa13" containerName="cleanup"
Nov 02 08:00:22.402946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.402933    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93831e27-0aad-40de-babf-ce03c15daa13" containerName="cleanup"
Nov 02 08:00:22.403037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.403027    4898 cpu_manager.go:414] "containerMap: removing container" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" containerName="cleanup"
Nov 02 08:00:22.403116 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.403107    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" containerName="cleanup"
Nov 02 08:00:22.403210 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.403201    4898 cpu_manager.go:414] "containerMap: removing container" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" containerName="cleanup"
Nov 02 08:00:22.403280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.403271    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" containerName="cleanup"
Nov 02 08:00:22.403363 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.403355    4898 cpu_manager.go:414] "containerMap: removing container" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" containerName="cleanup"
Nov 02 08:00:22.403434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.403425    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" containerName="cleanup"
Nov 02 08:00:22.403518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.403506    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" containerName="cleanup"
Nov 02 08:00:22.403621 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.403604    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" containerName="cleanup"
Nov 02 08:00:22.403723 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.403710    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" containerName="cleanup"
Nov 02 08:00:22.403811 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.403800    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" containerName="cleanup"
Nov 02 08:00:22.403923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.403912    4898 cpu_manager.go:414] "containerMap: removing container" podUID="99f50886-1073-441c-ad74-1dcc66339adc" containerName="cleanup"
Nov 02 08:00:22.404019 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.404008    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f50886-1073-441c-ad74-1dcc66339adc" containerName="cleanup"
Nov 02 08:00:22.404096 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.404087    4898 cpu_manager.go:414] "containerMap: removing container" podUID="42866b28-00e6-4781-a563-b42446b3e432" containerName="cleanup"
Nov 02 08:00:22.404179 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.404168    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42866b28-00e6-4781-a563-b42446b3e432" containerName="cleanup"
Nov 02 08:00:22.404263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.404253    4898 cpu_manager.go:414] "containerMap: removing container" podUID="42866b28-00e6-4781-a563-b42446b3e432" containerName="cleanup"
Nov 02 08:00:22.404339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.404329    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42866b28-00e6-4781-a563-b42446b3e432" containerName="cleanup"
Nov 02 08:00:22.404424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.404414    4898 cpu_manager.go:414] "containerMap: removing container" podUID="42866b28-00e6-4781-a563-b42446b3e432" containerName="cleanup"
Nov 02 08:00:22.404500 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.404488    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42866b28-00e6-4781-a563-b42446b3e432" containerName="cleanup"
Nov 02 08:00:22.404805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.404783    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="93831e27-0aad-40de-babf-ce03c15daa13" containerName="cleanup"
Nov 02 08:00:22.404955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.404941    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f50886-1073-441c-ad74-1dcc66339adc" containerName="cleanup"
Nov 02 08:00:22.405055 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405043    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed459624-7446-4717-b618-ec235eb93ce7" containerName="cleanup"
Nov 02 08:00:22.405136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405127    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="93831e27-0aad-40de-babf-ce03c15daa13" containerName="cleanup"
Nov 02 08:00:22.405208 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405199    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f50886-1073-441c-ad74-1dcc66339adc" containerName="cleanup"
Nov 02 08:00:22.405273 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405265    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" containerName="cleanup"
Nov 02 08:00:22.405339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405331    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" containerName="cleanup"
Nov 02 08:00:22.405410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405402    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" containerName="cleanup"
Nov 02 08:00:22.405474 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405466    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" containerName="cleanup"
Nov 02 08:00:22.405549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405540    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="93831e27-0aad-40de-babf-ce03c15daa13" containerName="cleanup"
Nov 02 08:00:22.405611 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405603    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" containerName="cleanup"
Nov 02 08:00:22.405679 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405670    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="42866b28-00e6-4781-a563-b42446b3e432" containerName="cleanup"
Nov 02 08:00:22.405746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405738    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="42866b28-00e6-4781-a563-b42446b3e432" containerName="cleanup"
Nov 02 08:00:22.405813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405805    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="42866b28-00e6-4781-a563-b42446b3e432" containerName="cleanup"
Nov 02 08:00:22.405875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405866    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" containerName="cleanup"
Nov 02 08:00:22.405964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.405955    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" containerName="cleanup"
Nov 02 08:00:22.406051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.406042    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" containerName="cleanup"
Nov 02 08:00:22.406124 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.406115    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed459624-7446-4717-b618-ec235eb93ce7" containerName="cleanup"
Nov 02 08:00:22.406194 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.406185    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" containerName="cleanup"
Nov 02 08:00:22.434787 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.432210    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"]
Nov 02 08:00:22.434787 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.432310    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.441979 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.438462    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr" event={"ID":"92df2ddf-95bc-4fbd-ada9-df92a5ca62cd","Type":"ContainerDied","Data":"81192508cd816bb49c66e06d2c9ad6a67b4ba7709104ce27875a76507b2471ba"}
Nov 02 08:00:22.441979 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.438529    4898 scope.go:122] "RemoveContainer" containerID="6ccac0c86144a6888e75b0906542d91b3dbfffaf1f843b5805c541d0a9007bce"
Nov 02 08:00:22.441979 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.438654    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"
Nov 02 08:00:22.452122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.447994    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-bb4907-openshift-default-dockercfg-m45r2\""
Nov 02 08:00:22.452830 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.452794    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl" event={"ID":"93831e27-0aad-40de-babf-ce03c15daa13","Type":"ContainerDied","Data":"29648d352321e5b27815144930e13bd86ad34a49bb0d3244cf262609e1c16de7"}
Nov 02 08:00:22.452961 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.452943    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"
Nov 02 08:00:22.463460 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.463280    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4" event={"ID":"ed459624-7446-4717-b618-ec235eb93ce7","Type":"ContainerDied","Data":"b7af7b4c3dbbbda70f3249293e957ed5a3a13453d4bd84ac3a8bffe0b7485b26"}
Nov 02 08:00:22.463460 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.463390    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"
Nov 02 08:00:22.552710 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.552680    4898 scope.go:122] "RemoveContainer" containerID="b40e2dbff54cfb09dae97aad6764960e2c0b87a1d43af89482703d31ece65987"
Nov 02 08:00:22.582099 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.582065    4898 scope.go:122] "RemoveContainer" containerID="c3dc24545d0ab279a2ab8dd1524cd690083f9cd2d191e5caa60da25125524575"
Nov 02 08:00:22.631449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.631362    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-podinfo\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.631449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.631410    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-envoy\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.631449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.631444    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-token\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.631742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.631498    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvfl7\" (UniqueName: \"kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-kube-api-access-nvfl7\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.631742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.631527    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-data\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.631742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.631552    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istiod-ca-cert\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.631742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.631573    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-certs\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.631742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.631635    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-socket\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.631742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.631659    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-credential-socket\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.733063 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.733016    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-podinfo\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.733166 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.733075    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-envoy\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.733166 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.733128    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-token\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.733280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.733201    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfl7\" (UniqueName: \"kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-kube-api-access-nvfl7\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.733280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.733242    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-data\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.733388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.733283    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istiod-ca-cert\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.733388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.733316    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-certs\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.733501 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.733413    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-socket\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.733501 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.733443    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-credential-socket\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.734293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.733882    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-data\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.734847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.734805    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-credential-socket\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.735018 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.734913    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-socket\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.735331 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.735296    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istiod-ca-cert\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.735682 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.735657    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-certs\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.741062 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.737179    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-podinfo\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.741062 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.737384    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-envoy\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.765279 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.765245    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfl7\" (UniqueName: \"kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-kube-api-access-nvfl7\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.769503 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.769468    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-token\") pod \"e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") " pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.826112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.819987    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"]
Nov 02 08:00:22.826112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.824241    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9db802-29893440-xh6q4"]
Nov 02 08:00:22.836141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.836109    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"
Nov 02 08:00:22.849943 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.849921    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:22.862478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.862452    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"]
Nov 02 08:00:22.862784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.862758    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"
Nov 02 08:00:22.867139 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.867120    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893440-g2qfl"]
Nov 02 08:00:22.881262 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.881232    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"
Nov 02 08:00:22.907688 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.907658    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"]
Nov 02 08:00:22.912968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.912925    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0fbc17-29893440-hb8jr"]
Nov 02 08:00:22.936112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.936073    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/516fa8a4-8039-4ca0-ac0c-83817acc91c3-kube-api-access-rf2ff\" (UniqueName: \"kubernetes.io/projected/516fa8a4-8039-4ca0-ac0c-83817acc91c3-kube-api-access-rf2ff\") pod \"516fa8a4-8039-4ca0-ac0c-83817acc91c3\" (UID: \"516fa8a4-8039-4ca0-ac0c-83817acc91c3\") "
Nov 02 08:00:22.937399 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.937371    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/516fa8a4-8039-4ca0-ac0c-83817acc91c3-maas-api-service-ca" pod "516fa8a4-8039-4ca0-ac0c-83817acc91c3" (UID: "516fa8a4-8039-4ca0-ac0c-83817acc91c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:22.937541 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.937435    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/516fa8a4-8039-4ca0-ac0c-83817acc91c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/516fa8a4-8039-4ca0-ac0c-83817acc91c3-maas-api-service-ca\") pod \"516fa8a4-8039-4ca0-ac0c-83817acc91c3\" (UID: \"516fa8a4-8039-4ca0-ac0c-83817acc91c3\") "
Nov 02 08:00:22.938117 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.938089    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/516fa8a4-8039-4ca0-ac0c-83817acc91c3-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:22.938875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:22.938843    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/516fa8a4-8039-4ca0-ac0c-83817acc91c3-kube-api-access-rf2ff" pod "516fa8a4-8039-4ca0-ac0c-83817acc91c3" (UID: "516fa8a4-8039-4ca0-ac0c-83817acc91c3"). InnerVolumeSpecName "kube-api-access-rf2ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:23.043799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.041901    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-maas-api-service-ca\") pod \"bbfacf71-eed8-4fde-96ef-c2e0c909fa30\" (UID: \"bbfacf71-eed8-4fde-96ef-c2e0c909fa30\") "
Nov 02 08:00:23.043799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.042054    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-maas-api-service-ca\") pod \"490dc0b1-9ca5-4f40-8639-e7480c7cce8d\" (UID: \"490dc0b1-9ca5-4f40-8639-e7480c7cce8d\") "
Nov 02 08:00:23.043799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.042141    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-kube-api-access-mcknz\" (UniqueName: \"kubernetes.io/projected/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-kube-api-access-mcknz\") pod \"bbfacf71-eed8-4fde-96ef-c2e0c909fa30\" (UID: \"bbfacf71-eed8-4fde-96ef-c2e0c909fa30\") "
Nov 02 08:00:23.043799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.042325    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-kube-api-access-km2bj\" (UniqueName: \"kubernetes.io/projected/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-kube-api-access-km2bj\") pod \"490dc0b1-9ca5-4f40-8639-e7480c7cce8d\" (UID: \"490dc0b1-9ca5-4f40-8639-e7480c7cce8d\") "
Nov 02 08:00:23.043799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.042433    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-maas-api-service-ca" pod "490dc0b1-9ca5-4f40-8639-e7480c7cce8d" (UID: "490dc0b1-9ca5-4f40-8639-e7480c7cce8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:23.043799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.042767    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-maas-api-service-ca" pod "bbfacf71-eed8-4fde-96ef-c2e0c909fa30" (UID: "bbfacf71-eed8-4fde-96ef-c2e0c909fa30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:00:23.043799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.042941    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:23.043799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.042960    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf2ff\" (UniqueName: \"kubernetes.io/projected/516fa8a4-8039-4ca0-ac0c-83817acc91c3-kube-api-access-rf2ff\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:23.043799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.042973    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:23.045735 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.045707    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-kube-api-access-km2bj" pod "490dc0b1-9ca5-4f40-8639-e7480c7cce8d" (UID: "490dc0b1-9ca5-4f40-8639-e7480c7cce8d"). InnerVolumeSpecName "kube-api-access-km2bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:23.049164 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.049137    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-kube-api-access-mcknz" pod "bbfacf71-eed8-4fde-96ef-c2e0c909fa30" (UID: "bbfacf71-eed8-4fde-96ef-c2e0c909fa30"). InnerVolumeSpecName "kube-api-access-mcknz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:23.144778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.144746    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcknz\" (UniqueName: \"kubernetes.io/projected/bbfacf71-eed8-4fde-96ef-c2e0c909fa30-kube-api-access-mcknz\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:23.144778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.144777    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km2bj\" (UniqueName: \"kubernetes.io/projected/490dc0b1-9ca5-4f40-8639-e7480c7cce8d-kube-api-access-km2bj\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:23.173875 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:00:23.173845    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f1801e_b54e_4616_bb4c_e279bf1c8ee5.slice/crio-21336fdf20a0cc39036aac2f727ae906431e24226a14eb59d31275c15a8ad900 WatchSource:0}: Error finding container 21336fdf20a0cc39036aac2f727ae906431e24226a14eb59d31275c15a8ad900: Status 404 returned error can't find the container with id 21336fdf20a0cc39036aac2f727ae906431e24226a14eb59d31275c15a8ad900
Nov 02 08:00:23.175405 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.175380    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"]
Nov 02 08:00:23.177194 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.177162    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:23.177265 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.177223    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:23.177265 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.177253    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:23.467651 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.467565    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" path="/var/lib/kubelet/pods/92df2ddf-95bc-4fbd-ada9-df92a5ca62cd/volumes"
Nov 02 08:00:23.482398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.482360    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93831e27-0aad-40de-babf-ce03c15daa13" path="/var/lib/kubelet/pods/93831e27-0aad-40de-babf-ce03c15daa13/volumes"
Nov 02 08:00:23.483411 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.483383    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed459624-7446-4717-b618-ec235eb93ce7" path="/var/lib/kubelet/pods/ed459624-7446-4717-b618-ec235eb93ce7/volumes"
Nov 02 08:00:23.484285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.484263    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb" path="/var/lib/kubelet/pods/f2dd6fad-2ea4-499e-9b3c-0f65fad9cedb/volumes"
Nov 02 08:00:23.509578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.509540    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk" event={"ID":"516fa8a4-8039-4ca0-ac0c-83817acc91c3","Type":"ContainerDied","Data":"97667419c7dc68569cff132309cd006bbc7b0cc7b038ced35285f346390eef92"}
Nov 02 08:00:23.509712 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.509632    4898 scope.go:122] "RemoveContainer" containerID="5056eb36ed95d930c1da24f19660f1465ceba47fcdec6ae51ea04ebd4638d6d9"
Nov 02 08:00:23.510000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.509802    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"
Nov 02 08:00:23.513966 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.513882    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"]
Nov 02 08:00:23.516463 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516440    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" containerName="cleanup"
Nov 02 08:00:23.516463 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516463    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" containerName="cleanup"
Nov 02 08:00:23.516620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516479    4898 cpu_manager.go:414] "containerMap: removing container" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" containerName="cleanup"
Nov 02 08:00:23.516620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516487    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" containerName="cleanup"
Nov 02 08:00:23.516620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516503    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" containerName="cleanup"
Nov 02 08:00:23.516620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516510    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" containerName="cleanup"
Nov 02 08:00:23.516620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516533    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" containerName="cleanup"
Nov 02 08:00:23.516620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516540    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" containerName="cleanup"
Nov 02 08:00:23.516620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516573    4898 cpu_manager.go:414] "containerMap: removing container" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" containerName="cleanup"
Nov 02 08:00:23.516620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516580    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" containerName="cleanup"
Nov 02 08:00:23.516620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516592    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ed459624-7446-4717-b618-ec235eb93ce7" containerName="cleanup"
Nov 02 08:00:23.516620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516600    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed459624-7446-4717-b618-ec235eb93ce7" containerName="cleanup"
Nov 02 08:00:23.516620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516614    4898 cpu_manager.go:414] "containerMap: removing container" podUID="93831e27-0aad-40de-babf-ce03c15daa13" containerName="cleanup"
Nov 02 08:00:23.516620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516621    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93831e27-0aad-40de-babf-ce03c15daa13" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516631    4898 cpu_manager.go:414] "containerMap: removing container" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516638    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516650    4898 cpu_manager.go:414] "containerMap: removing container" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516657    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516931    4898 cpu_manager.go:414] "containerMap: removing container" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516945    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516969    4898 cpu_manager.go:414] "containerMap: removing container" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516976    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516991    4898 cpu_manager.go:414] "containerMap: removing container" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.516997    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517028    4898 cpu_manager.go:414] "containerMap: removing container" podUID="99f50886-1073-441c-ad74-1dcc66339adc" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517035    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f50886-1073-441c-ad74-1dcc66339adc" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517045    4898 cpu_manager.go:414] "containerMap: removing container" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517051    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" containerName="cleanup"
Nov 02 08:00:23.517213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517219    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" containerName="cleanup"
Nov 02 08:00:23.518094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517235    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" containerName="cleanup"
Nov 02 08:00:23.518094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517244    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" containerName="cleanup"
Nov 02 08:00:23.518094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517256    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" containerName="cleanup"
Nov 02 08:00:23.518094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517268    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e59499e-9863-4849-bc47-e7a91d1fcf7a" containerName="cleanup"
Nov 02 08:00:23.518094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517277    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce754bc4-4ab3-4b5b-8cb0-d35d080413bd" containerName="cleanup"
Nov 02 08:00:23.518094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517286    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" containerName="cleanup"
Nov 02 08:00:23.518094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517295    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed459624-7446-4717-b618-ec235eb93ce7" containerName="cleanup"
Nov 02 08:00:23.518094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517303    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f50886-1073-441c-ad74-1dcc66339adc" containerName="cleanup"
Nov 02 08:00:23.518094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517314    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" containerName="cleanup"
Nov 02 08:00:23.518094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517323    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" containerName="cleanup"
Nov 02 08:00:23.518094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.517333    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="92df2ddf-95bc-4fbd-ada9-df92a5ca62cd" containerName="cleanup"
Nov 02 08:00:23.518853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.518722    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"
Nov 02 08:00:23.523879 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.523858    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"
Nov 02 08:00:23.528217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.528180    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" event={"ID":"09f1801e-b54e-4616-bb4c-e279bf1c8ee5","Type":"ContainerStarted","Data":"dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003"}
Nov 02 08:00:23.528319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.528222    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" event={"ID":"09f1801e-b54e-4616-bb4c-e279bf1c8ee5","Type":"ContainerStarted","Data":"21336fdf20a0cc39036aac2f727ae906431e24226a14eb59d31275c15a8ad900"}
Nov 02 08:00:23.528319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.528246    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4" event={"ID":"bbfacf71-eed8-4fde-96ef-c2e0c909fa30","Type":"ContainerDied","Data":"49695756af242a65c9e7f11e1d80aaabc29fda99a9bb95dc38f83090f9b65f64"}
Nov 02 08:00:23.528319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.528275    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc" event={"ID":"490dc0b1-9ca5-4f40-8639-e7480c7cce8d","Type":"ContainerDied","Data":"d1ace0b2217f423b437e3c27773f84f110dfbe15f290d09f9a4e3ba83df41aee"}
Nov 02 08:00:23.528445 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.528363    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"
Nov 02 08:00:23.528973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.528945    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"]
Nov 02 08:00:23.551184 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.546543    4898 scope.go:122] "RemoveContainer" containerID="aa597c37acd24f185cadb5d93239b1b1824facdd18f59505211db6364565159d"
Nov 02 08:00:23.551184 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.548553    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-736f7c-dockercfg-559zh\""
Nov 02 08:00:23.551184 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.548648    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-736f7c\""
Nov 02 08:00:23.564002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.563866    4898 scope.go:122] "RemoveContainer" containerID="c83136e26087d70488f63944a2ceb8bb28cc0f34810ae9b2bd55c64fe7670eae"
Nov 02 08:00:23.583973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.583833    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" podStartSLOduration=1.583814411 podStartE2EDuration="1.583814411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:23.578110895 +0000 UTC m=+2174.268437859" watchObservedRunningTime="2026-11-02 08:00:23.583814411 +0000 UTC m=+2174.274141376"
Nov 02 08:00:23.644038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.644002    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"]
Nov 02 08:00:23.655978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.655944    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c704b690-8c9a-4a5f-8651-848a5bbba59a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r\" (UID: \"c704b690-8c9a-4a5f-8651-848a5bbba59a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"
Nov 02 08:00:23.657187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.657158    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w58q2\" (UniqueName: \"kubernetes.io/projected/c704b690-8c9a-4a5f-8651-848a5bbba59a-kube-api-access-w58q2\") pod \"payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r\" (UID: \"c704b690-8c9a-4a5f-8651-848a5bbba59a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"
Nov 02 08:00:23.677706 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.677672    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893440-28szk"]
Nov 02 08:00:23.776133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.774315    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c704b690-8c9a-4a5f-8651-848a5bbba59a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r\" (UID: \"c704b690-8c9a-4a5f-8651-848a5bbba59a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"
Nov 02 08:00:23.776133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.774421    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w58q2\" (UniqueName: \"kubernetes.io/projected/c704b690-8c9a-4a5f-8651-848a5bbba59a-kube-api-access-w58q2\") pod \"payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r\" (UID: \"c704b690-8c9a-4a5f-8651-848a5bbba59a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"
Nov 02 08:00:23.776133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.775955    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c704b690-8c9a-4a5f-8651-848a5bbba59a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r\" (UID: \"c704b690-8c9a-4a5f-8651-848a5bbba59a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"
Nov 02 08:00:23.811742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.811707    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58q2\" (UniqueName: \"kubernetes.io/projected/c704b690-8c9a-4a5f-8651-848a5bbba59a-kube-api-access-w58q2\") pod \"payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r\" (UID: \"c704b690-8c9a-4a5f-8651-848a5bbba59a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"
Nov 02 08:00:23.851156 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.851124    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:23.865679 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.865644    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"
Nov 02 08:00:23.883461 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.880525    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"]
Nov 02 08:00:23.890811 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.890750    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893440-4vfnc"]
Nov 02 08:00:23.898810 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.898767    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"]
Nov 02 08:00:23.904986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:23.904958    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893440-pkkz4"]
Nov 02 08:00:24.205530 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:00:24.205490    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc704b690_8c9a_4a5f_8651_848a5bbba59a.slice/crio-e0650cab8991840918ee1eaf255d8d219c8d1c0ffb8370721110725ef0ae809c WatchSource:0}: Error finding container e0650cab8991840918ee1eaf255d8d219c8d1c0ffb8370721110725ef0ae809c: Status 404 returned error can't find the container with id e0650cab8991840918ee1eaf255d8d219c8d1c0ffb8370721110725ef0ae809c
Nov 02 08:00:24.206584 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:24.206560    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"]
Nov 02 08:00:24.581142 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:24.581038    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r" event={"ID":"c704b690-8c9a-4a5f-8651-848a5bbba59a","Type":"ContainerStarted","Data":"bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449"}
Nov 02 08:00:24.581718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:24.581687    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r" event={"ID":"c704b690-8c9a-4a5f-8651-848a5bbba59a","Type":"ContainerStarted","Data":"e0650cab8991840918ee1eaf255d8d219c8d1c0ffb8370721110725ef0ae809c"}
Nov 02 08:00:24.582956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:24.582914    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"
Nov 02 08:00:24.862157 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:24.862072    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:24.910879 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:24.910824    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r" podStartSLOduration=1.910810323 podStartE2EDuration="1.910810323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:24.630175413 +0000 UTC m=+2175.320502386" watchObservedRunningTime="2026-11-02 08:00:24.910810323 +0000 UTC m=+2175.601137286"
Nov 02 08:00:25.555909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:25.538790    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" path="/var/lib/kubelet/pods/490dc0b1-9ca5-4f40-8639-e7480c7cce8d/volumes"
Nov 02 08:00:25.555909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:25.539477    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="516fa8a4-8039-4ca0-ac0c-83817acc91c3" path="/var/lib/kubelet/pods/516fa8a4-8039-4ca0-ac0c-83817acc91c3/volumes"
Nov 02 08:00:25.555909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:25.540074    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" path="/var/lib/kubelet/pods/bbfacf71-eed8-4fde-96ef-c2e0c909fa30/volumes"
Nov 02 08:00:25.629064 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:25.628681    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:25.629064 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:25.628764    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:00:28.082405 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.082368    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"]
Nov 02 08:00:28.086911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.086868    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" containerName="cleanup"
Nov 02 08:00:28.087073 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.087059    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" containerName="cleanup"
Nov 02 08:00:28.087370 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.087357    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="490dc0b1-9ca5-4f40-8639-e7480c7cce8d" containerName="cleanup"
Nov 02 08:00:28.087471 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.087462    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbfacf71-eed8-4fde-96ef-c2e0c909fa30" containerName="cleanup"
Nov 02 08:00:28.131676 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.125283    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"
Nov 02 08:00:28.159914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.140919    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"]
Nov 02 08:00:28.213036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.192221    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxsgp\" (UniqueName: \"kubernetes.io/projected/8f53dcda-e118-4c9e-ba95-097a44061d18-kube-api-access-nxsgp\") pod \"payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb\" (UID: \"8f53dcda-e118-4c9e-ba95-097a44061d18\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"
Nov 02 08:00:28.233077 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.233037    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f53dcda-e118-4c9e-ba95-097a44061d18-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb\" (UID: \"8f53dcda-e118-4c9e-ba95-097a44061d18\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"
Nov 02 08:00:28.237998 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.214856    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-bb4907-dockercfg-4tbvs\""
Nov 02 08:00:28.304913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.214163    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-bb4907\""
Nov 02 08:00:28.335263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.335163    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxsgp\" (UniqueName: \"kubernetes.io/projected/8f53dcda-e118-4c9e-ba95-097a44061d18-kube-api-access-nxsgp\") pod \"payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb\" (UID: \"8f53dcda-e118-4c9e-ba95-097a44061d18\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"
Nov 02 08:00:28.335263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.335257    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f53dcda-e118-4c9e-ba95-097a44061d18-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb\" (UID: \"8f53dcda-e118-4c9e-ba95-097a44061d18\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"
Nov 02 08:00:28.336157 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.336134    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f53dcda-e118-4c9e-ba95-097a44061d18-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb\" (UID: \"8f53dcda-e118-4c9e-ba95-097a44061d18\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"
Nov 02 08:00:28.362641 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.362591    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxsgp\" (UniqueName: \"kubernetes.io/projected/8f53dcda-e118-4c9e-ba95-097a44061d18-kube-api-access-nxsgp\") pod \"payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb\" (UID: \"8f53dcda-e118-4c9e-ba95-097a44061d18\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"
Nov 02 08:00:28.500089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.500037    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"
Nov 02 08:00:28.838799 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:00:28.838725    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f53dcda_e118_4c9e_ba95_097a44061d18.slice/crio-899bf1fcbdc38e95e0a89bf2f68b575e9cfdeb39b8bd65ae4de9dde93fbd7d5a WatchSource:0}: Error finding container 899bf1fcbdc38e95e0a89bf2f68b575e9cfdeb39b8bd65ae4de9dde93fbd7d5a: Status 404 returned error can't find the container with id 899bf1fcbdc38e95e0a89bf2f68b575e9cfdeb39b8bd65ae4de9dde93fbd7d5a
Nov 02 08:00:28.839321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:28.839298    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"]
Nov 02 08:00:29.684139 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:29.684091    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb" event={"ID":"8f53dcda-e118-4c9e-ba95-097a44061d18","Type":"ContainerStarted","Data":"79ab56cf2fcc620dde84884af9626041ea66e51e89b65190c50a8ddb7b8e0092"}
Nov 02 08:00:29.684587 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:29.684175    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"
Nov 02 08:00:29.684587 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:29.684194    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb" event={"ID":"8f53dcda-e118-4c9e-ba95-097a44061d18","Type":"ContainerStarted","Data":"899bf1fcbdc38e95e0a89bf2f68b575e9cfdeb39b8bd65ae4de9dde93fbd7d5a"}
Nov 02 08:00:30.226141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.226095    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj" podUID="bf5f52e9-580d-4f4d-b272-463a3e22b277" containerName="main" containerID="cri-o://398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c" gracePeriod=45
Nov 02 08:00:30.543337 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.543307    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 08:00:30.592405 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.592353    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb" podStartSLOduration=2.592337874 podStartE2EDuration="2.592337874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:29.744358628 +0000 UTC m=+2180.434685592" watchObservedRunningTime="2026-11-02 08:00:30.592337874 +0000 UTC m=+2181.282664879"
Nov 02 08:00:30.679922 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.679855    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-dshm\") pod \"bf5f52e9-580d-4f4d-b272-463a3e22b277\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") "
Nov 02 08:00:30.680100 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.680046    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-tmp-dir\") pod \"bf5f52e9-580d-4f4d-b272-463a3e22b277\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") "
Nov 02 08:00:30.680100 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.680089    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-model-cache\") pod \"bf5f52e9-580d-4f4d-b272-463a3e22b277\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") "
Nov 02 08:00:30.680257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.680139    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-home\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-home\") pod \"bf5f52e9-580d-4f4d-b272-463a3e22b277\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") "
Nov 02 08:00:30.680257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.680192    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf5f52e9-580d-4f4d-b272-463a3e22b277-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5f52e9-580d-4f4d-b272-463a3e22b277-tls-certs\") pod \"bf5f52e9-580d-4f4d-b272-463a3e22b277\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") "
Nov 02 08:00:30.680257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.680221    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5f52e9-580d-4f4d-b272-463a3e22b277-kube-api-access-2txx2\" (UniqueName: \"kubernetes.io/projected/bf5f52e9-580d-4f4d-b272-463a3e22b277-kube-api-access-2txx2\") pod \"bf5f52e9-580d-4f4d-b272-463a3e22b277\" (UID: \"bf5f52e9-580d-4f4d-b272-463a3e22b277\") "
Nov 02 08:00:30.680528 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.680343    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-tmp-dir" pod "bf5f52e9-580d-4f4d-b272-463a3e22b277" (UID: "bf5f52e9-580d-4f4d-b272-463a3e22b277"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:30.680528 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.680442    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-model-cache" pod "bf5f52e9-580d-4f4d-b272-463a3e22b277" (UID: "bf5f52e9-580d-4f4d-b272-463a3e22b277"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:30.680956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.680933    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:30.681108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.680958    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:30.681108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.681063    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-home" pod "bf5f52e9-580d-4f4d-b272-463a3e22b277" (UID: "bf5f52e9-580d-4f4d-b272-463a3e22b277"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:30.682845 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.682806    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-dshm" pod "bf5f52e9-580d-4f4d-b272-463a3e22b277" (UID: "bf5f52e9-580d-4f4d-b272-463a3e22b277"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:00:30.682947 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.682913    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5f52e9-580d-4f4d-b272-463a3e22b277-kube-api-access-2txx2" pod "bf5f52e9-580d-4f4d-b272-463a3e22b277" (UID: "bf5f52e9-580d-4f4d-b272-463a3e22b277"). InnerVolumeSpecName "kube-api-access-2txx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:00:30.683254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.683233    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf5f52e9-580d-4f4d-b272-463a3e22b277-tls-certs" pod "bf5f52e9-580d-4f4d-b272-463a3e22b277" (UID: "bf5f52e9-580d-4f4d-b272-463a3e22b277"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:00:30.694660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.694634    4898 generic.go:356] "Generic (PLEG): container finished" podID="bf5f52e9-580d-4f4d-b272-463a3e22b277" containerID="398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c" exitCode=0
Nov 02 08:00:30.695042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.694688    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"
Nov 02 08:00:30.695042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.694717    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj" event={"ID":"bf5f52e9-580d-4f4d-b272-463a3e22b277","Type":"ContainerDied","Data":"398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c"}
Nov 02 08:00:30.695042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.694764    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj" event={"ID":"bf5f52e9-580d-4f4d-b272-463a3e22b277","Type":"ContainerDied","Data":"4571ca4d5d541bc374248d2e3f5fa6def2e7bcd5251507916cc04275bdd6cdd2"}
Nov 02 08:00:30.695042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.694786    4898 scope.go:122] "RemoveContainer" containerID="398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c"
Nov 02 08:00:30.719831 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.719811    4898 scope.go:122] "RemoveContainer" containerID="398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c"
Nov 02 08:00:30.720174 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:30.720152    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c\": container with ID starting with 398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c not found: ID does not exist" containerID="398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c"
Nov 02 08:00:30.720256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.720187    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c"} err="failed to get container status \"398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c\": rpc error: code = NotFound desc = could not find container \"398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c\": container with ID starting with 398ca31377c2862aa618eed57f12cae8410443128acc47be49234b3ddfeb6d8c not found: ID does not exist"
Nov 02 08:00:30.762697 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.762621    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"]
Nov 02 08:00:30.770325 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.770297    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-3575b9/auth-test-model-3575b9-kserve-6698cf7fdf-gqvmj"]
Nov 02 08:00:30.782723 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.782688    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:30.782723 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.782719    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5f52e9-580d-4f4d-b272-463a3e22b277-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:30.782941 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.782735    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2txx2\" (UniqueName: \"kubernetes.io/projected/bf5f52e9-580d-4f4d-b272-463a3e22b277-kube-api-access-2txx2\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:30.782941 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:30.782750    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5f52e9-580d-4f4d-b272-463a3e22b277-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:00:31.334303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.334270    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"]
Nov 02 08:00:31.334968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.334951    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bf5f52e9-580d-4f4d-b272-463a3e22b277" containerName="main"
Nov 02 08:00:31.334968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.334968    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5f52e9-580d-4f4d-b272-463a3e22b277" containerName="main"
Nov 02 08:00:31.335121 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.335112    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5f52e9-580d-4f4d-b272-463a3e22b277" containerName="main"
Nov 02 08:00:31.339591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.339575    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"
Nov 02 08:00:31.360026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.360001    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"]
Nov 02 08:00:31.366057 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.366032    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-7a8bb6\""
Nov 02 08:00:31.366232 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.366048    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-7a8bb6-dockercfg-lrlls\""
Nov 02 08:00:31.493916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.486630    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzvcx\" (UniqueName: \"kubernetes.io/projected/2950e4a5-896e-4388-ba7e-accccdf2153a-kube-api-access-jzvcx\") pod \"payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl\" (UID: \"2950e4a5-896e-4388-ba7e-accccdf2153a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"
Nov 02 08:00:31.493916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.486739    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2950e4a5-896e-4388-ba7e-accccdf2153a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl\" (UID: \"2950e4a5-896e-4388-ba7e-accccdf2153a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"
Nov 02 08:00:31.613813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.595953    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzvcx\" (UniqueName: \"kubernetes.io/projected/2950e4a5-896e-4388-ba7e-accccdf2153a-kube-api-access-jzvcx\") pod \"payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl\" (UID: \"2950e4a5-896e-4388-ba7e-accccdf2153a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"
Nov 02 08:00:31.613813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.596058    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2950e4a5-896e-4388-ba7e-accccdf2153a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl\" (UID: \"2950e4a5-896e-4388-ba7e-accccdf2153a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"
Nov 02 08:00:31.613813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.597148    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2950e4a5-896e-4388-ba7e-accccdf2153a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl\" (UID: \"2950e4a5-896e-4388-ba7e-accccdf2153a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"
Nov 02 08:00:31.626941 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.624358    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5f52e9-580d-4f4d-b272-463a3e22b277" path="/var/lib/kubelet/pods/bf5f52e9-580d-4f4d-b272-463a3e22b277/volumes"
Nov 02 08:00:31.626941 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.625036    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"]
Nov 02 08:00:31.632243 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.632198    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzvcx\" (UniqueName: \"kubernetes.io/projected/2950e4a5-896e-4388-ba7e-accccdf2153a-kube-api-access-jzvcx\") pod \"payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl\" (UID: \"2950e4a5-896e-4388-ba7e-accccdf2153a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"
Nov 02 08:00:31.639234 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.639211    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"]
Nov 02 08:00:31.639397 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.639381    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"
Nov 02 08:00:31.654139 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.654114    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"
Nov 02 08:00:31.697425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.697383    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfc2h\" (UniqueName: \"kubernetes.io/projected/2e973ac4-18ee-48dd-8b9c-6d25ac250822-kube-api-access-sfc2h\") pod \"payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8\" (UID: \"2e973ac4-18ee-48dd-8b9c-6d25ac250822\") " pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"
Nov 02 08:00:31.697937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.697532    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e973ac4-18ee-48dd-8b9c-6d25ac250822-plugins-config-volume\") pod \"payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8\" (UID: \"2e973ac4-18ee-48dd-8b9c-6d25ac250822\") " pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"
Nov 02 08:00:31.799705 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.799122    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e973ac4-18ee-48dd-8b9c-6d25ac250822-plugins-config-volume\") pod \"payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8\" (UID: \"2e973ac4-18ee-48dd-8b9c-6d25ac250822\") " pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"
Nov 02 08:00:31.799705 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.799430    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfc2h\" (UniqueName: \"kubernetes.io/projected/2e973ac4-18ee-48dd-8b9c-6d25ac250822-kube-api-access-sfc2h\") pod \"payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8\" (UID: \"2e973ac4-18ee-48dd-8b9c-6d25ac250822\") " pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"
Nov 02 08:00:31.800539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.800511    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e973ac4-18ee-48dd-8b9c-6d25ac250822-plugins-config-volume\") pod \"payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8\" (UID: \"2e973ac4-18ee-48dd-8b9c-6d25ac250822\") " pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"
Nov 02 08:00:31.823646 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.823612    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfc2h\" (UniqueName: \"kubernetes.io/projected/2e973ac4-18ee-48dd-8b9c-6d25ac250822-kube-api-access-sfc2h\") pod \"payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8\" (UID: \"2e973ac4-18ee-48dd-8b9c-6d25ac250822\") " pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"
Nov 02 08:00:31.963933 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:00:31.963870    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2950e4a5_896e_4388_ba7e_accccdf2153a.slice/crio-4e52bf90e29bfefd80f30726d5c5da1cf94f2e5de6e293d43808ab584a6235dc WatchSource:0}: Error finding container 4e52bf90e29bfefd80f30726d5c5da1cf94f2e5de6e293d43808ab584a6235dc: Status 404 returned error can't find the container with id 4e52bf90e29bfefd80f30726d5c5da1cf94f2e5de6e293d43808ab584a6235dc
Nov 02 08:00:31.965512 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.965486    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"]
Nov 02 08:00:31.968965 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:31.968934    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"
Nov 02 08:00:32.303918 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:00:32.303859    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e973ac4_18ee_48dd_8b9c_6d25ac250822.slice/crio-121947efd076931a9b838e58cac9e6ad28bd01905d35c1db37a046eec9ae7e7a WatchSource:0}: Error finding container 121947efd076931a9b838e58cac9e6ad28bd01905d35c1db37a046eec9ae7e7a: Status 404 returned error can't find the container with id 121947efd076931a9b838e58cac9e6ad28bd01905d35c1db37a046eec9ae7e7a
Nov 02 08:00:32.306402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:32.306377    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"]
Nov 02 08:00:32.716250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:32.716201    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8" event={"ID":"2e973ac4-18ee-48dd-8b9c-6d25ac250822","Type":"ContainerStarted","Data":"6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3"}
Nov 02 08:00:32.716732 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:32.716269    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8" event={"ID":"2e973ac4-18ee-48dd-8b9c-6d25ac250822","Type":"ContainerStarted","Data":"121947efd076931a9b838e58cac9e6ad28bd01905d35c1db37a046eec9ae7e7a"}
Nov 02 08:00:32.716732 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:32.716301    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"
Nov 02 08:00:32.717950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:32.717922    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl" event={"ID":"2950e4a5-896e-4388-ba7e-accccdf2153a","Type":"ContainerStarted","Data":"62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4"}
Nov 02 08:00:32.718114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:32.717975    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl" event={"ID":"2950e4a5-896e-4388-ba7e-accccdf2153a","Type":"ContainerStarted","Data":"4e52bf90e29bfefd80f30726d5c5da1cf94f2e5de6e293d43808ab584a6235dc"}
Nov 02 08:00:32.718114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:32.718003    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"
Nov 02 08:00:32.760616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:32.760565    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8" podStartSLOduration=1.760552884 podStartE2EDuration="1.760552884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:32.758244276 +0000 UTC m=+2183.448571241" watchObservedRunningTime="2026-11-02 08:00:32.760552884 +0000 UTC m=+2183.450879848"
Nov 02 08:00:36.635759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:36.635728    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"
Nov 02 08:00:36.693642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:36.693552    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl" podStartSLOduration=5.693529627 podStartE2EDuration="5.693529627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:32.803055891 +0000 UTC m=+2183.493382856" watchObservedRunningTime="2026-11-02 08:00:36.693529627 +0000 UTC m=+2187.383856593"
Nov 02 08:00:39.546545 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.546509    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"]
Nov 02 08:00:39.551699 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.551679    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.597031 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.590484    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-643355-openshift-default-dockercfg-7ch4b\""
Nov 02 08:00:39.608206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.608176    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"]
Nov 02 08:00:39.708866 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.708828    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-token\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.709093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.708958    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-certs\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.709093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.709027    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-podinfo\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.709093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.709050    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-credential-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.709275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.709163    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-envoy\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.709275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.709195    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmwp\" (UniqueName: \"kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-kube-api-access-sbmwp\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.709378 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.709331    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3341400d-3694-48ec-86e8-e0ed90f5776b-istiod-ca-cert\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.709442 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.709373    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-data\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.709502 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.709481    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.813624 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.813526    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-podinfo\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.828027 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.827987    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-credential-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.845068 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.845026    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-envoy\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.845353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.845330    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmwp\" (UniqueName: \"kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-kube-api-access-sbmwp\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.845612 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.845590    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3341400d-3694-48ec-86e8-e0ed90f5776b-istiod-ca-cert\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.850014 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.849983    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-data\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.850283 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.850263    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.850436 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.850418    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-token\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.865915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.850613    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-certs\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.962818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.857333    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-certs\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.962818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.864374    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-credential-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.962818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.876316    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-data\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.962818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.876565    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-socket\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.962818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.958797    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3341400d-3694-48ec-86e8-e0ed90f5776b-istiod-ca-cert\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.962818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.959135    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-envoy\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:39.976189 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:39.969611    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-podinfo\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:40.013192 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:40.011903    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmwp\" (UniqueName: \"kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-kube-api-access-sbmwp\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:40.016228 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:40.016195    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-token\") pod \"e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") " pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:40.209949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:40.209909    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:40.592409 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:00:40.592371    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3341400d_3694_48ec_86e8_e0ed90f5776b.slice/crio-7b0216bac6f83ecd0ecbc7c7abb62065a3aa80f3ec6f364ae5c3f98c48b163d1 WatchSource:0}: Error finding container 7b0216bac6f83ecd0ecbc7c7abb62065a3aa80f3ec6f364ae5c3f98c48b163d1: Status 404 returned error can't find the container with id 7b0216bac6f83ecd0ecbc7c7abb62065a3aa80f3ec6f364ae5c3f98c48b163d1
Nov 02 08:00:40.595498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:40.595460    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:40.595608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:40.595534    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:40.595608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:40.595564    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:40.595755 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:40.595729    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"]
Nov 02 08:00:40.700013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:40.699982    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"
Nov 02 08:00:41.032498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:41.031659    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" event={"ID":"3341400d-3694-48ec-86e8-e0ed90f5776b","Type":"ContainerStarted","Data":"cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00"}
Nov 02 08:00:41.032498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:41.031710    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" event={"ID":"3341400d-3694-48ec-86e8-e0ed90f5776b","Type":"ContainerStarted","Data":"7b0216bac6f83ecd0ecbc7c7abb62065a3aa80f3ec6f364ae5c3f98c48b163d1"}
Nov 02 08:00:41.211914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:41.210262    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:41.227913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:41.224066    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 02 08:00:41.227913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:41.224164    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 02 08:00:42.216350 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:42.216317    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:42.278775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:42.278714    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" podStartSLOduration=3.278697433 podStartE2EDuration="3.278697433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:00:41.097182643 +0000 UTC m=+2191.787509610" watchObservedRunningTime="2026-11-02 08:00:42.278697433 +0000 UTC m=+2192.969024457"
Nov 02 08:00:43.049016 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:43.048981    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:43.050141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:43.050112    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:00:43.730242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:43.730207    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"
Nov 02 08:00:43.731083 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:43.730281    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"
Nov 02 08:00:44.993261 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:00:44.993204    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:00:46.080405 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:46.080370    4898 generic.go:356] "Generic (PLEG): container finished" podID="9609fcde-fbb5-409b-82bf-78eaec870453" containerID="9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e" exitCode=0
Nov 02 08:00:46.081032 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:46.080452    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" event={"ID":"9609fcde-fbb5-409b-82bf-78eaec870453","Type":"ContainerDied","Data":"9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e"}
Nov 02 08:00:46.081117 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:46.081037    4898 scope.go:122] "RemoveContainer" containerID="9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e"
Nov 02 08:00:46.083252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:46.083212    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:46.083380 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:46.083313    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:46.083380 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:46.083359    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:00:47.090590 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:47.090552    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" event={"ID":"9609fcde-fbb5-409b-82bf-78eaec870453","Type":"ContainerStarted","Data":"514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a"}
Nov 02 08:00:47.535876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:47.535838    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 08:00:47.536085 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:47.536025    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 02 08:00:47.536151 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:47.536076    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 02 08:00:48.535850 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:48.535815    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 02 08:00:48.536303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:48.535867    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 02 08:00:49.535749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:49.535713    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 02 08:00:49.535957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:49.535784    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 02 08:00:50.535223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:50.535183    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 02 08:00:50.535461 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:50.535247    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 02 08:00:51.536399 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:51.536300    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Nov 02 08:00:51.537092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:51.537057    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Nov 02 08:00:52.555919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:52.555461    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 08:00:52.559438 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:52.557206    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 08:00:52.559598 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:52.559581    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 08:00:58.651277 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:58.651234    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"]
Nov 02 08:00:58.657255 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:58.657230    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"
Nov 02 08:00:58.685553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:58.685507    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"]
Nov 02 08:00:58.690552 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:58.690524    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ea9556\""
Nov 02 08:00:58.690866 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:58.690820    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ea9556-dockercfg-jgxbp\""
Nov 02 08:00:58.838816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:58.791301    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961445c8-a7a3-464c-846c-fb355e661f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk\" (UID: \"961445c8-a7a3-464c-846c-fb355e661f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"
Nov 02 08:00:58.838816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:58.837980    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqk2n\" (UniqueName: \"kubernetes.io/projected/961445c8-a7a3-464c-846c-fb355e661f6e-kube-api-access-fqk2n\") pod \"payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk\" (UID: \"961445c8-a7a3-464c-846c-fb355e661f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"
Nov 02 08:00:58.940227 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:58.940119    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961445c8-a7a3-464c-846c-fb355e661f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk\" (UID: \"961445c8-a7a3-464c-846c-fb355e661f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"
Nov 02 08:00:58.940227 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:58.940186    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqk2n\" (UniqueName: \"kubernetes.io/projected/961445c8-a7a3-464c-846c-fb355e661f6e-kube-api-access-fqk2n\") pod \"payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk\" (UID: \"961445c8-a7a3-464c-846c-fb355e661f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"
Nov 02 08:00:58.941547 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:58.941474    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961445c8-a7a3-464c-846c-fb355e661f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk\" (UID: \"961445c8-a7a3-464c-846c-fb355e661f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"
Nov 02 08:00:58.964475 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:58.964445    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqk2n\" (UniqueName: \"kubernetes.io/projected/961445c8-a7a3-464c-846c-fb355e661f6e-kube-api-access-fqk2n\") pod \"payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk\" (UID: \"961445c8-a7a3-464c-846c-fb355e661f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"
Nov 02 08:00:58.974301 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:58.974269    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"
Nov 02 08:00:59.284280 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:00:59.284239    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961445c8_a7a3_464c_846c_fb355e661f6e.slice/crio-c633efe76c760766054d398e3b99c6e55248717ccae2296b8ca993081dcf4b0b WatchSource:0}: Error finding container c633efe76c760766054d398e3b99c6e55248717ccae2296b8ca993081dcf4b0b: Status 404 returned error can't find the container with id c633efe76c760766054d398e3b99c6e55248717ccae2296b8ca993081dcf4b0b
Nov 02 08:00:59.285065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:00:59.285035    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"]
Nov 02 08:01:00.209857 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:00.209812    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk" event={"ID":"961445c8-a7a3-464c-846c-fb355e661f6e","Type":"ContainerStarted","Data":"9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822"}
Nov 02 08:01:00.210317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:00.209880    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk" event={"ID":"961445c8-a7a3-464c-846c-fb355e661f6e","Type":"ContainerStarted","Data":"c633efe76c760766054d398e3b99c6e55248717ccae2296b8ca993081dcf4b0b"}
Nov 02 08:01:00.210317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:00.209951    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"
Nov 02 08:01:00.277910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:00.276901    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk" podStartSLOduration=2.276861073 podStartE2EDuration="2.276861073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:01:00.251013164 +0000 UTC m=+2210.941340152" watchObservedRunningTime="2026-11-02 08:01:00.276861073 +0000 UTC m=+2210.967188037"
Nov 02 08:01:11.222948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:11.222874    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"
Nov 02 08:01:14.990623 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:01:14.990575    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:01:39.611504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:39.611465    4898 generic.go:356] "Generic (PLEG): container finished" podID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerID="dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003" exitCode=0
Nov 02 08:01:39.611970 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:39.611531    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" event={"ID":"09f1801e-b54e-4616-bb4c-e279bf1c8ee5","Type":"ContainerDied","Data":"dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003"}
Nov 02 08:01:39.612029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:39.612018    4898 scope.go:122] "RemoveContainer" containerID="dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003"
Nov 02 08:01:39.614291 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:39.614251    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:01:39.614395 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:39.614352    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:01:39.614436 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:39.614399    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:01:39.944010 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:39.943971    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"]
Nov 02 08:01:39.988189 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:39.988137    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.000006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:39.989938    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-podinfo\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.000006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:39.999817    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzcml\" (UniqueName: \"kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-kube-api-access-zzcml\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.000006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:39.999867    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-token\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.000006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.000003    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-data\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.000388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.000068    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istiod-ca-cert\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.000388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.000142    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-socket\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.000388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.000206    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-credential-socket\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.000388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.000234    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-certs\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.000388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.000296    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-envoy\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.007750 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.007721    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"]
Nov 02 08:01:40.021383 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.020250    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2dde84-openshift-default-dockercfg-jxgwx\""
Nov 02 08:01:40.107548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.107458    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-data\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.107838 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.107814    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istiod-ca-cert\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.108036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.108019    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-socket\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.108193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.108177    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-credential-socket\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.108330 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.108313    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-certs\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.108488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.108472    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-envoy\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.108659 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.108643    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-podinfo\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.108879 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.108855    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzcml\" (UniqueName: \"kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-kube-api-access-zzcml\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.109289 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.109266    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-token\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.109439 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.109423    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-socket\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.109831 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.109809    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-data\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.110098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.110072    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-credential-socket\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.110348 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.110324    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-certs\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.116745 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.116723    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-envoy\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.117106 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.117084    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istiod-ca-cert\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.120504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.120483    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-podinfo\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.132904 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.132872    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-token\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.134776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.134751    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzcml\" (UniqueName: \"kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-kube-api-access-zzcml\") pod \"e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") " pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.412276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.412245    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:40.673831 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.672732    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:01:40.710742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.710702    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" event={"ID":"09f1801e-b54e-4616-bb4c-e279bf1c8ee5","Type":"ContainerStarted","Data":"76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f"}
Nov 02 08:01:40.831227 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.831193    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"]
Nov 02 08:01:40.843408 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.843372    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:01:40.843493 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.843454    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:01:40.843493 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.843485    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:01:40.850728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.850701    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:01:40.852199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.852176    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 08:01:40.852312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:40.852219    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 08:01:41.731109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:41.731065    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" event={"ID":"8d3b67b4-66e9-4f49-942d-61b4b632ac82","Type":"ContainerStarted","Data":"5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963"}
Nov 02 08:01:41.731578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:41.731164    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" event={"ID":"8d3b67b4-66e9-4f49-942d-61b4b632ac82","Type":"ContainerStarted","Data":"9b3306756cee18e89e3dcf7fbec6f15aba2eec04ee40f094c4d152b3d2567f89"}
Nov 02 08:01:41.804954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:41.804871    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" podStartSLOduration=2.804854727 podStartE2EDuration="2.804854727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:01:41.804231289 +0000 UTC m=+2252.494558254" watchObservedRunningTime="2026-11-02 08:01:41.804854727 +0000 UTC m=+2252.495181765"
Nov 02 08:01:41.850333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:41.850297    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 08:01:41.850529 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:41.850358    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 08:01:42.412971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:42.412924    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:42.418991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:42.418965    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:42.738965 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:42.738867    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:42.739781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:42.739755    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:01:42.851317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:42.851064    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 08:01:42.851317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:42.851123    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 08:01:43.851138 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:43.851105    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 08:01:43.851531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:43.851168    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 08:01:44.851124 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:44.851090    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 08:01:44.851320 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:44.851152    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 08:01:45.016611 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:01:45.016572    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:01:45.850504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:45.850471    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 08:01:45.850779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:45.850523    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 08:01:46.856911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:46.856307    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 08:01:46.856911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:46.856377    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 08:01:47.896457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:47.896425    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:01:47.904216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:47.904176    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:01:47.906968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:47.906941    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:01:59.091047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:59.090996    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:01:59.091600 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:01:59.091059    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:02:02.007161 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:02.006855    4898 generic.go:356] "Generic (PLEG): container finished" podID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerID="cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00" exitCode=0
Nov 02 08:02:02.007161 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:02.007005    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" event={"ID":"3341400d-3694-48ec-86e8-e0ed90f5776b","Type":"ContainerDied","Data":"cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00"}
Nov 02 08:02:02.007790 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:02.007554    4898 scope.go:122] "RemoveContainer" containerID="cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00"
Nov 02 08:02:02.028133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:02.010647    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:02:02.028133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:02.010740    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:02:02.028133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:02.026051    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:02:03.016962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:03.016928    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" event={"ID":"3341400d-3694-48ec-86e8-e0ed90f5776b","Type":"ContainerStarted","Data":"d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69"}
Nov 02 08:02:03.210641 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:03.210603    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:02:03.210842 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:03.210818    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 02 08:02:03.210940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:03.210913    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 02 08:02:04.210672 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:04.210636    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 02 08:02:04.211191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:04.210693    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 02 08:02:05.210487 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:05.210449    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 02 08:02:05.210674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:05.210516    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 02 08:02:06.211109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:06.211076    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 02 08:02:06.211520 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:06.211142    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 02 08:02:07.210840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:07.210806    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 02 08:02:07.211045 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:07.210868    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 02 08:02:08.210556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:08.210514    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 02 08:02:08.211089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:08.210572    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 02 08:02:09.318047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.318012    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:02:09.318990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.318966    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:02:09.320483 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.320462    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:02:09.385396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.385365    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"]
Nov 02 08:02:09.390049 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.390019    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:09.408395 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.408367    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"]
Nov 02 08:02:09.409731 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.409707    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-plugins-e2e-extmod-b-643355\""
Nov 02 08:02:09.409731 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.409707    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"openshift-service-ca.crt\""
Nov 02 08:02:09.409971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.409788    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"routing-overlay\""
Nov 02 08:02:09.409971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.409806    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"kube-root-ca.crt\""
Nov 02 08:02:09.427258 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.427226    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-e2e--9fb8cda0-dockercfg-h9ktn\""
Nov 02 08:02:09.427564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.427518    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-bfdb1bbb5e7a\""
Nov 02 08:02:09.518384 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.518348    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:09.518579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.518402    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:09.518579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.518467    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:09.518579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.518498    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6bw\" (UniqueName: \"kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-kube-api-access-xv6bw\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:09.622173 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.622076    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:09.623065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.623037    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:09.623333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.623313    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:09.623506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.623488    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6bw\" (UniqueName: \"kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-kube-api-access-xv6bw\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:09.657937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.654092    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-plugins-e2e-extmod-b-643355\""
Nov 02 08:02:09.657937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.654180    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"routing-overlay\""
Nov 02 08:02:09.657937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.654244    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-bfdb1bbb5e7a\""
Nov 02 08:02:09.657937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.655750    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:09.657937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.656444    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:09.675145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.675106    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"kube-root-ca.crt\""
Nov 02 08:02:09.711973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.709719    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:09.711973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.709984    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"openshift-service-ca.crt\""
Nov 02 08:02:09.731185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:09.731150    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6bw\" (UniqueName: \"kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-kube-api-access-xv6bw\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:10.020884 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:10.020855    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-e2e--9fb8cda0-dockercfg-h9ktn\""
Nov 02 08:02:10.027371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:10.027344    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:10.332245 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:02:10.332214    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b81fa1_5a4b_439b_baae_d654f7c0f373.slice/crio-6e5f71c0733b9b6db2ac2dd3d0d270e2048e487bb91fe8e5e9bc00d6aec6a8c8 WatchSource:0}: Error finding container 6e5f71c0733b9b6db2ac2dd3d0d270e2048e487bb91fe8e5e9bc00d6aec6a8c8: Status 404 returned error can't find the container with id 6e5f71c0733b9b6db2ac2dd3d0d270e2048e487bb91fe8e5e9bc00d6aec6a8c8
Nov 02 08:02:10.336504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:10.336477    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"]
Nov 02 08:02:11.136714 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:11.136677    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t" event={"ID":"e0b81fa1-5a4b-439b-baae-d654f7c0f373","Type":"ContainerStarted","Data":"f0d9f7cf6f584fb85ff1f1150ee4d978c95a2bb2fdd43cd9a5f835c3358bb7fa"}
Nov 02 08:02:11.136976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:11.136736    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t" event={"ID":"e0b81fa1-5a4b-439b-baae-d654f7c0f373","Type":"ContainerStarted","Data":"6e5f71c0733b9b6db2ac2dd3d0d270e2048e487bb91fe8e5e9bc00d6aec6a8c8"}
Nov 02 08:02:11.136976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:11.136881    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:11.184837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:11.184768    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t" podStartSLOduration=2.184749847 podStartE2EDuration="2.184749847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:02:11.179276544 +0000 UTC m=+2281.869603509" watchObservedRunningTime="2026-11-02 08:02:11.184749847 +0000 UTC m=+2281.875076811"
Nov 02 08:02:14.991296 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:02:14.991255    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:02:22.154845 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:22.154768    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:02:25.883109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:25.883072    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"]
Nov 02 08:02:25.888520 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:25.888490    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"
Nov 02 08:02:25.908127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:25.908087    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"]
Nov 02 08:02:25.913726 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:25.913698    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-71a41c\""
Nov 02 08:02:25.913879 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:25.913759    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-71a41c-dockercfg-kjs2s\""
Nov 02 08:02:25.955635 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:25.955599    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f1541ea-41ea-4c23-8f9c-9279f17592ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z\" (UID: \"8f1541ea-41ea-4c23-8f9c-9279f17592ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"
Nov 02 08:02:25.955807 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:25.955645    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwz98\" (UniqueName: \"kubernetes.io/projected/8f1541ea-41ea-4c23-8f9c-9279f17592ca-kube-api-access-rwz98\") pod \"payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z\" (UID: \"8f1541ea-41ea-4c23-8f9c-9279f17592ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"
Nov 02 08:02:25.971543 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:25.971501    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"]
Nov 02 08:02:25.976253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:25.976227    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"
Nov 02 08:02:26.002300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.002075    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"]
Nov 02 08:02:26.186919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.178094    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3566a46-846d-46a7-86e5-26159d70d0dc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv\" (UID: \"c3566a46-846d-46a7-86e5-26159d70d0dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"
Nov 02 08:02:26.186919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.178309    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q526\" (UniqueName: \"kubernetes.io/projected/c3566a46-846d-46a7-86e5-26159d70d0dc-kube-api-access-4q526\") pod \"payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv\" (UID: \"c3566a46-846d-46a7-86e5-26159d70d0dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"
Nov 02 08:02:26.186919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.178360    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f1541ea-41ea-4c23-8f9c-9279f17592ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z\" (UID: \"8f1541ea-41ea-4c23-8f9c-9279f17592ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"
Nov 02 08:02:26.186919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.178411    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwz98\" (UniqueName: \"kubernetes.io/projected/8f1541ea-41ea-4c23-8f9c-9279f17592ca-kube-api-access-rwz98\") pod \"payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z\" (UID: \"8f1541ea-41ea-4c23-8f9c-9279f17592ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"
Nov 02 08:02:26.203177 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.197119    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f1541ea-41ea-4c23-8f9c-9279f17592ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z\" (UID: \"8f1541ea-41ea-4c23-8f9c-9279f17592ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"
Nov 02 08:02:26.266914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.266738    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwz98\" (UniqueName: \"kubernetes.io/projected/8f1541ea-41ea-4c23-8f9c-9279f17592ca-kube-api-access-rwz98\") pod \"payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z\" (UID: \"8f1541ea-41ea-4c23-8f9c-9279f17592ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"
Nov 02 08:02:26.279783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.279741    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q526\" (UniqueName: \"kubernetes.io/projected/c3566a46-846d-46a7-86e5-26159d70d0dc-kube-api-access-4q526\") pod \"payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv\" (UID: \"c3566a46-846d-46a7-86e5-26159d70d0dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"
Nov 02 08:02:26.279981 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.279939    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3566a46-846d-46a7-86e5-26159d70d0dc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv\" (UID: \"c3566a46-846d-46a7-86e5-26159d70d0dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"
Nov 02 08:02:26.280710 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.280673    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3566a46-846d-46a7-86e5-26159d70d0dc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv\" (UID: \"c3566a46-846d-46a7-86e5-26159d70d0dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"
Nov 02 08:02:26.305117 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.305079    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q526\" (UniqueName: \"kubernetes.io/projected/c3566a46-846d-46a7-86e5-26159d70d0dc-kube-api-access-4q526\") pod \"payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv\" (UID: \"c3566a46-846d-46a7-86e5-26159d70d0dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"
Nov 02 08:02:26.505424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.505341    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"
Nov 02 08:02:26.600393 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.599150    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"
Nov 02 08:02:26.844595 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:02:26.844565    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1541ea_41ea_4c23_8f9c_9279f17592ca.slice/crio-685f46fd75a23c11e3ffd4a0a989a6d2ed0f3a944773d2fb49878e7cd4162a56 WatchSource:0}: Error finding container 685f46fd75a23c11e3ffd4a0a989a6d2ed0f3a944773d2fb49878e7cd4162a56: Status 404 returned error can't find the container with id 685f46fd75a23c11e3ffd4a0a989a6d2ed0f3a944773d2fb49878e7cd4162a56
Nov 02 08:02:26.844869 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.844847    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"]
Nov 02 08:02:26.935770 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:02:26.935729    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3566a46_846d_46a7_86e5_26159d70d0dc.slice/crio-89a5199c23a78069bd7051c24d9c8d89a16843fa84d6fe8e3923dfb99f1b2fcf WatchSource:0}: Error finding container 89a5199c23a78069bd7051c24d9c8d89a16843fa84d6fe8e3923dfb99f1b2fcf: Status 404 returned error can't find the container with id 89a5199c23a78069bd7051c24d9c8d89a16843fa84d6fe8e3923dfb99f1b2fcf
Nov 02 08:02:26.937233 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:26.937207    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"]
Nov 02 08:02:27.422091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:27.422052    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv" event={"ID":"c3566a46-846d-46a7-86e5-26159d70d0dc","Type":"ContainerStarted","Data":"8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e"}
Nov 02 08:02:27.422312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:27.422114    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv" event={"ID":"c3566a46-846d-46a7-86e5-26159d70d0dc","Type":"ContainerStarted","Data":"89a5199c23a78069bd7051c24d9c8d89a16843fa84d6fe8e3923dfb99f1b2fcf"}
Nov 02 08:02:27.422312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:27.422170    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"
Nov 02 08:02:27.424010 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:27.423982    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z" event={"ID":"8f1541ea-41ea-4c23-8f9c-9279f17592ca","Type":"ContainerStarted","Data":"2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9"}
Nov 02 08:02:27.424153 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:27.424029    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z" event={"ID":"8f1541ea-41ea-4c23-8f9c-9279f17592ca","Type":"ContainerStarted","Data":"685f46fd75a23c11e3ffd4a0a989a6d2ed0f3a944773d2fb49878e7cd4162a56"}
Nov 02 08:02:27.424153 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:27.424054    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"
Nov 02 08:02:27.463988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:27.463876    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv" podStartSLOduration=2.463859416 podStartE2EDuration="2.463859416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:02:27.463116924 +0000 UTC m=+2298.153443886" watchObservedRunningTime="2026-11-02 08:02:27.463859416 +0000 UTC m=+2298.154186383"
Nov 02 08:02:27.518556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:27.518491    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z" podStartSLOduration=2.518470884 podStartE2EDuration="2.518470884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:02:27.517334594 +0000 UTC m=+2298.207661560" watchObservedRunningTime="2026-11-02 08:02:27.518470884 +0000 UTC m=+2298.208797850"
Nov 02 08:02:33.236207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:33.236169    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"]
Nov 02 08:02:33.236768 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:33.236389    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerName="istio-proxy" containerID="cri-o://35729fa9dfe327ec7989e8008e873b3a48ad3cd09925adfc5836d62a30be3654" gracePeriod=30
Nov 02 08:02:34.449913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:34.449851    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"]
Nov 02 08:02:34.450331 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:34.450065    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy" containerID="cri-o://6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e" gracePeriod=30
Nov 02 08:02:38.225179 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.225145    4898 patch_prober.go:29] interesting pod/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:02:38.225583 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.225207    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:02:38.436644 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.435681    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"
Nov 02 08:02:38.440103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.440080    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"
Nov 02 08:02:38.529975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.529769    4898 generic.go:356] "Generic (PLEG): container finished" podID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerID="35729fa9dfe327ec7989e8008e873b3a48ad3cd09925adfc5836d62a30be3654" exitCode=0
Nov 02 08:02:38.529975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.529803    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" event={"ID":"b4441a3e-ee8c-4af8-94f7-e9a4114ad603","Type":"ContainerDied","Data":"35729fa9dfe327ec7989e8008e873b3a48ad3cd09925adfc5836d62a30be3654"}
Nov 02 08:02:38.529975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.529870    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc" event={"ID":"b4441a3e-ee8c-4af8-94f7-e9a4114ad603","Type":"ContainerDied","Data":"87e33f3d490556bc529735d869a0311e56107ae0f6e96d21e2ad18d9349c462a"}
Nov 02 08:02:38.529975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.529910    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87e33f3d490556bc529735d869a0311e56107ae0f6e96d21e2ad18d9349c462a"
Nov 02 08:02:38.549387 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.549353    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 08:02:38.570708 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.570673    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-kube-api-access-h2rbm\" (UniqueName: \"kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-kube-api-access-h2rbm\") pod \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") "
Nov 02 08:02:38.571021 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.570996    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-podinfo\") pod \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") "
Nov 02 08:02:38.571210 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.571191    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-credential-socket\") pod \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") "
Nov 02 08:02:38.571313 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.571286    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istiod-ca-cert\") pod \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") "
Nov 02 08:02:38.571397 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.571333    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-socket\") pod \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") "
Nov 02 08:02:38.571484 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.571405    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-envoy\") pod \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") "
Nov 02 08:02:38.571484 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.571440    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-token\" (UniqueName: \"kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-token\") pod \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") "
Nov 02 08:02:38.571641 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.571542    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-data\") pod \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") "
Nov 02 08:02:38.571875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.571621    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-certs\") pod \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\" (UID: \"b4441a3e-ee8c-4af8-94f7-e9a4114ad603\") "
Nov 02 08:02:38.572764 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.572739    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-certs" pod "b4441a3e-ee8c-4af8-94f7-e9a4114ad603" (UID: "b4441a3e-ee8c-4af8-94f7-e9a4114ad603"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:38.573432 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.573405    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-credential-socket" pod "b4441a3e-ee8c-4af8-94f7-e9a4114ad603" (UID: "b4441a3e-ee8c-4af8-94f7-e9a4114ad603"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:38.573840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.573814    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-socket" pod "b4441a3e-ee8c-4af8-94f7-e9a4114ad603" (UID: "b4441a3e-ee8c-4af8-94f7-e9a4114ad603"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:38.574215 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.574190    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istiod-ca-cert" pod "b4441a3e-ee8c-4af8-94f7-e9a4114ad603" (UID: "b4441a3e-ee8c-4af8-94f7-e9a4114ad603"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:02:38.575462 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.575434    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-data" pod "b4441a3e-ee8c-4af8-94f7-e9a4114ad603" (UID: "b4441a3e-ee8c-4af8-94f7-e9a4114ad603"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:38.576816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.576675    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-envoy" pod "b4441a3e-ee8c-4af8-94f7-e9a4114ad603" (UID: "b4441a3e-ee8c-4af8-94f7-e9a4114ad603"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:38.577346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.577312    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-podinfo" pod "b4441a3e-ee8c-4af8-94f7-e9a4114ad603" (UID: "b4441a3e-ee8c-4af8-94f7-e9a4114ad603"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:02:38.577712 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.577590    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-token" pod "b4441a3e-ee8c-4af8-94f7-e9a4114ad603" (UID: "b4441a3e-ee8c-4af8-94f7-e9a4114ad603"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:02:38.581469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.581444    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-kube-api-access-h2rbm" pod "b4441a3e-ee8c-4af8-94f7-e9a4114ad603" (UID: "b4441a3e-ee8c-4af8-94f7-e9a4114ad603"). InnerVolumeSpecName "kube-api-access-h2rbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:02:38.673645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.673608    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:38.673645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.673638    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:38.673645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.673651    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:38.673938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.673663    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:38.673938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.673675    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:38.673938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.673687    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:38.673938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.673707    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:38.673938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.673720    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2rbm\" (UniqueName: \"kubernetes.io/projected/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-kube-api-access-h2rbm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:38.673938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:38.673732    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4441a3e-ee8c-4af8-94f7-e9a4114ad603-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:39.540007 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.539981    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"
Nov 02 08:02:39.610870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.610836    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"]
Nov 02 08:02:39.618032 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.618002    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-lgzxc"]
Nov 02 08:02:39.729226 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.729168    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 08:02:39.786675 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.786640    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-data\") pod \"b8912595-8874-4a75-b536-eeecdbfcc40a\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") "
Nov 02 08:02:39.786850 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.786701    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-socket\") pod \"b8912595-8874-4a75-b536-eeecdbfcc40a\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") "
Nov 02 08:02:39.786850 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.786757    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-token\" (UniqueName: \"kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-token\") pod \"b8912595-8874-4a75-b536-eeecdbfcc40a\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") "
Nov 02 08:02:39.786850 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.786801    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-certs\") pod \"b8912595-8874-4a75-b536-eeecdbfcc40a\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") "
Nov 02 08:02:39.787044 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.786963    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-envoy\") pod \"b8912595-8874-4a75-b536-eeecdbfcc40a\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") "
Nov 02 08:02:39.787044 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.786982    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-data" pod "b8912595-8874-4a75-b536-eeecdbfcc40a" (UID: "b8912595-8874-4a75-b536-eeecdbfcc40a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:39.787044 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.786988    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-socket" pod "b8912595-8874-4a75-b536-eeecdbfcc40a" (UID: "b8912595-8874-4a75-b536-eeecdbfcc40a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:39.787044 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.787038    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8912595-8874-4a75-b536-eeecdbfcc40a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8912595-8874-4a75-b536-eeecdbfcc40a-istiod-ca-cert\") pod \"b8912595-8874-4a75-b536-eeecdbfcc40a\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") "
Nov 02 08:02:39.787266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.787128    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-credential-socket\") pod \"b8912595-8874-4a75-b536-eeecdbfcc40a\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") "
Nov 02 08:02:39.787266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.787185    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-certs" pod "b8912595-8874-4a75-b536-eeecdbfcc40a" (UID: "b8912595-8874-4a75-b536-eeecdbfcc40a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:39.787266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.787253    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-kube-api-access-spqft\" (UniqueName: \"kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-kube-api-access-spqft\") pod \"b8912595-8874-4a75-b536-eeecdbfcc40a\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") "
Nov 02 08:02:39.787443 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.787282    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-podinfo\") pod \"b8912595-8874-4a75-b536-eeecdbfcc40a\" (UID: \"b8912595-8874-4a75-b536-eeecdbfcc40a\") "
Nov 02 08:02:39.787443 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.787409    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-credential-socket" pod "b8912595-8874-4a75-b536-eeecdbfcc40a" (UID: "b8912595-8874-4a75-b536-eeecdbfcc40a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:39.787557 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.787432    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8912595-8874-4a75-b536-eeecdbfcc40a-istiod-ca-cert" pod "b8912595-8874-4a75-b536-eeecdbfcc40a" (UID: "b8912595-8874-4a75-b536-eeecdbfcc40a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:02:39.787927 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.787874    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:39.787927 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.787922    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:39.788123 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.787936    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:39.788123 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.787951    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8912595-8874-4a75-b536-eeecdbfcc40a-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:39.788123 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.787963    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:39.789793 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.789764    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-token" pod "b8912595-8874-4a75-b536-eeecdbfcc40a" (UID: "b8912595-8874-4a75-b536-eeecdbfcc40a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:02:39.789956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.789849    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-envoy" pod "b8912595-8874-4a75-b536-eeecdbfcc40a" (UID: "b8912595-8874-4a75-b536-eeecdbfcc40a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:02:39.790021 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.789996    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-kube-api-access-spqft" pod "b8912595-8874-4a75-b536-eeecdbfcc40a" (UID: "b8912595-8874-4a75-b536-eeecdbfcc40a"). InnerVolumeSpecName "kube-api-access-spqft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:02:39.790675 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.790651    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-podinfo" pod "b8912595-8874-4a75-b536-eeecdbfcc40a" (UID: "b8912595-8874-4a75-b536-eeecdbfcc40a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:02:39.890765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.890679    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spqft\" (UniqueName: \"kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-kube-api-access-spqft\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:39.890765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.890739    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:39.890765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.891477    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:39.890765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:39.891504    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8912595-8874-4a75-b536-eeecdbfcc40a-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:02:40.549789 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:40.549755    4898 generic.go:356] "Generic (PLEG): container finished" podID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerID="6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e" exitCode=0
Nov 02 08:02:40.550316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:40.549815    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"
Nov 02 08:02:40.550316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:40.549848    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" event={"ID":"b8912595-8874-4a75-b536-eeecdbfcc40a","Type":"ContainerDied","Data":"6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e"}
Nov 02 08:02:40.550316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:40.549937    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x" event={"ID":"b8912595-8874-4a75-b536-eeecdbfcc40a","Type":"ContainerDied","Data":"cb46723595005ad1f2d00d0f41f0686fff69f361ba929c4dc2507f7345cca9ac"}
Nov 02 08:02:40.550316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:40.549964    4898 scope.go:122] "RemoveContainer" containerID="6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e"
Nov 02 08:02:40.562961 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:40.562878    4898 scope.go:122] "RemoveContainer" containerID="6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e"
Nov 02 08:02:40.563286 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:02:40.563266    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e\": container with ID starting with 6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e not found: ID does not exist" containerID="6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e"
Nov 02 08:02:40.563368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:40.563297    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e"} err="failed to get container status \"6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e\": rpc error: code = NotFound desc = could not find container \"6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e\": container with ID starting with 6d96aba74ec90b273d0afe0cb95b049a580088febb0d89828060eb185fcc8e4e not found: ID does not exist"
Nov 02 08:02:40.610761 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:40.610722    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"]
Nov 02 08:02:40.622648 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:40.622614    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-m4f8x"]
Nov 02 08:02:41.466448 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:41.466415    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" path="/var/lib/kubelet/pods/b4441a3e-ee8c-4af8-94f7-e9a4114ad603/volumes"
Nov 02 08:02:41.466945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:41.466930    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" path="/var/lib/kubelet/pods/b8912595-8874-4a75-b536-eeecdbfcc40a/volumes"
Nov 02 08:02:44.989661 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:02:44.989610    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:02:56.069860 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.069822    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"]
Nov 02 08:02:56.070556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.070537    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerName="istio-proxy"
Nov 02 08:02:56.070603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.070557    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerName="istio-proxy"
Nov 02 08:02:56.070603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.070579    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy"
Nov 02 08:02:56.070603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.070584    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy"
Nov 02 08:02:56.070726 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.070717    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4441a3e-ee8c-4af8-94f7-e9a4114ad603" containerName="istio-proxy"
Nov 02 08:02:56.070763 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.070729    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8912595-8874-4a75-b536-eeecdbfcc40a" containerName="istio-proxy"
Nov 02 08:02:56.076243 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.076224    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.120023 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.119984    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-7a8bb6-openshift-default-dockercfg-8lb7h\""
Nov 02 08:02:56.127338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.127304    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"]
Nov 02 08:02:56.168438 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.168410    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-podinfo\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.168604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.168444    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-certs\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.168604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.168477    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-data\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.168699 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.168611    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-socket\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.168699 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.168658    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9c024c-f87e-4232-8588-e48db8f85ba9-istiod-ca-cert\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.168820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.168759    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-envoy\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.168820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.168791    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-credential-socket\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.168947 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.168832    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-token\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.168947 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.168861    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zvmx\" (UniqueName: \"kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-kube-api-access-4zvmx\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.270399 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.270364    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-socket\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.270618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.270413    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9c024c-f87e-4232-8588-e48db8f85ba9-istiod-ca-cert\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.270618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.270463    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-envoy\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.270618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.270480    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-credential-socket\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.270920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.270609    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-token\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.270920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.270672    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zvmx\" (UniqueName: \"kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-kube-api-access-4zvmx\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.270920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.270809    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-podinfo\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.270920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.270845    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-certs\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.271159 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.270883    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-data\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.271159 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.270976    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-socket\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.271159 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.271009    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-credential-socket\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.271588 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.271566    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-data\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.271668 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.271644    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-certs\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.271871 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.271849    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9c024c-f87e-4232-8588-e48db8f85ba9-istiod-ca-cert\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.273087 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.273058    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-envoy\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.273643 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.273624    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-podinfo\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.295198 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.295160    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-token\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.295317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.295227    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zvmx\" (UniqueName: \"kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-kube-api-access-4zvmx\") pod \"e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") " pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.390838 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.390755    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:56.716753 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:02:56.716709    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9c024c_f87e_4232_8588_e48db8f85ba9.slice/crio-e798113d01978eba4f6bc7574843bc2d05fca14677d480e9f28d1a500081e49f WatchSource:0}: Error finding container e798113d01978eba4f6bc7574843bc2d05fca14677d480e9f28d1a500081e49f: Status 404 returned error can't find the container with id e798113d01978eba4f6bc7574843bc2d05fca14677d480e9f28d1a500081e49f
Nov 02 08:02:56.718868 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.718795    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"]
Nov 02 08:02:56.720452 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.720411    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:02:56.720521 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.720498    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:02:56.720565 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:56.720541    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:02:57.719298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:57.719259    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg" event={"ID":"4f9c024c-f87e-4232-8588-e48db8f85ba9","Type":"ContainerStarted","Data":"c688a8b311f44765014aea11de8e40fde3145fedf38ad339717294154599fd81"}
Nov 02 08:02:57.719969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:57.719314    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg" event={"ID":"4f9c024c-f87e-4232-8588-e48db8f85ba9","Type":"ContainerStarted","Data":"e798113d01978eba4f6bc7574843bc2d05fca14677d480e9f28d1a500081e49f"}
Nov 02 08:02:57.784348 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:57.784268    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg" podStartSLOduration=1.7842444830000002 podStartE2EDuration="1.784244483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:02:57.778497572 +0000 UTC m=+2328.468824537" watchObservedRunningTime="2026-11-02 08:02:57.784244483 +0000 UTC m=+2328.474571448"
Nov 02 08:02:58.391618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:58.391462    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:02:58.394067 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:58.394032    4898 patch_prober.go:29] interesting pod/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Nov 02 08:02:58.394207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:58.394082    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg" podUID="4f9c024c-f87e-4232-8588-e48db8f85ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Nov 02 08:02:59.391242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:59.391205    4898 patch_prober.go:29] interesting pod/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Nov 02 08:02:59.391642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:02:59.391265    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg" podUID="4f9c024c-f87e-4232-8588-e48db8f85ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Nov 02 08:03:00.391184 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:00.391147    4898 patch_prober.go:29] interesting pod/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Nov 02 08:03:00.391347 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:00.391202    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg" podUID="4f9c024c-f87e-4232-8588-e48db8f85ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Nov 02 08:03:00.802781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:00.802743    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"]
Nov 02 08:03:00.840380 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:00.840348    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"]
Nov 02 08:03:01.391165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:01.391118    4898 patch_prober.go:29] interesting pod/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Nov 02 08:03:01.391395 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:01.391181    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg" podUID="4f9c024c-f87e-4232-8588-e48db8f85ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Nov 02 08:03:02.393125 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:02.393081    4898 patch_prober.go:29] interesting pod/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Nov 02 08:03:02.393583 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:02.393152    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg" podUID="4f9c024c-f87e-4232-8588-e48db8f85ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Nov 02 08:03:03.396525 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:03.396487    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:03:03.397001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:03.396976    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:03:03.397703 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:03.397681    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:03:05.803771 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:05.803719    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8" podUID="2e973ac4-18ee-48dd-8b9c-6d25ac250822" containerName="payload-processing" containerID="cri-o://6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3" gracePeriod=25
Nov 02 08:03:05.841455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:05.841416    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl" podUID="2950e4a5-896e-4388-ba7e-accccdf2153a" containerName="payload-pre-processing" containerID="cri-o://62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4" gracePeriod=25
Nov 02 08:03:06.135288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.135257    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"
Nov 02 08:03:06.206396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.206355    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e973ac4-18ee-48dd-8b9c-6d25ac250822-kube-api-access-sfc2h\" (UniqueName: \"kubernetes.io/projected/2e973ac4-18ee-48dd-8b9c-6d25ac250822-kube-api-access-sfc2h\") pod \"2e973ac4-18ee-48dd-8b9c-6d25ac250822\" (UID: \"2e973ac4-18ee-48dd-8b9c-6d25ac250822\") "
Nov 02 08:03:06.206838 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.206814    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e973ac4-18ee-48dd-8b9c-6d25ac250822-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e973ac4-18ee-48dd-8b9c-6d25ac250822-plugins-config-volume\") pod \"2e973ac4-18ee-48dd-8b9c-6d25ac250822\" (UID: \"2e973ac4-18ee-48dd-8b9c-6d25ac250822\") "
Nov 02 08:03:06.208060 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.208035    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e973ac4-18ee-48dd-8b9c-6d25ac250822-plugins-config-volume" pod "2e973ac4-18ee-48dd-8b9c-6d25ac250822" (UID: "2e973ac4-18ee-48dd-8b9c-6d25ac250822"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:03:06.218410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.218371    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e973ac4-18ee-48dd-8b9c-6d25ac250822-kube-api-access-sfc2h" pod "2e973ac4-18ee-48dd-8b9c-6d25ac250822" (UID: "2e973ac4-18ee-48dd-8b9c-6d25ac250822"). InnerVolumeSpecName "kube-api-access-sfc2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:03:06.262611 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.262580    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"
Nov 02 08:03:06.308824 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.308791    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2950e4a5-896e-4388-ba7e-accccdf2153a-kube-api-access-jzvcx\" (UniqueName: \"kubernetes.io/projected/2950e4a5-896e-4388-ba7e-accccdf2153a-kube-api-access-jzvcx\") pod \"2950e4a5-896e-4388-ba7e-accccdf2153a\" (UID: \"2950e4a5-896e-4388-ba7e-accccdf2153a\") "
Nov 02 08:03:06.309052 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.308860    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2950e4a5-896e-4388-ba7e-accccdf2153a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2950e4a5-896e-4388-ba7e-accccdf2153a-plugins-config-volume\") pod \"2950e4a5-896e-4388-ba7e-accccdf2153a\" (UID: \"2950e4a5-896e-4388-ba7e-accccdf2153a\") "
Nov 02 08:03:06.309243 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.309219    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2950e4a5-896e-4388-ba7e-accccdf2153a-plugins-config-volume" pod "2950e4a5-896e-4388-ba7e-accccdf2153a" (UID: "2950e4a5-896e-4388-ba7e-accccdf2153a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:03:06.309382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.309369    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfc2h\" (UniqueName: \"kubernetes.io/projected/2e973ac4-18ee-48dd-8b9c-6d25ac250822-kube-api-access-sfc2h\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:06.309455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.309383    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2950e4a5-896e-4388-ba7e-accccdf2153a-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:06.309455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.309393    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e973ac4-18ee-48dd-8b9c-6d25ac250822-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:06.311044 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.311021    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2950e4a5-896e-4388-ba7e-accccdf2153a-kube-api-access-jzvcx" pod "2950e4a5-896e-4388-ba7e-accccdf2153a" (UID: "2950e4a5-896e-4388-ba7e-accccdf2153a"). InnerVolumeSpecName "kube-api-access-jzvcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:03:06.410577 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.410540    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzvcx\" (UniqueName: \"kubernetes.io/projected/2950e4a5-896e-4388-ba7e-accccdf2153a-kube-api-access-jzvcx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:06.814875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.814787    4898 generic.go:356] "Generic (PLEG): container finished" podID="2e973ac4-18ee-48dd-8b9c-6d25ac250822" containerID="6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3" exitCode=0
Nov 02 08:03:06.814875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.814852    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"
Nov 02 08:03:06.815387 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.814874    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8" event={"ID":"2e973ac4-18ee-48dd-8b9c-6d25ac250822","Type":"ContainerDied","Data":"6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3"}
Nov 02 08:03:06.815387 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.814964    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8" event={"ID":"2e973ac4-18ee-48dd-8b9c-6d25ac250822","Type":"ContainerDied","Data":"121947efd076931a9b838e58cac9e6ad28bd01905d35c1db37a046eec9ae7e7a"}
Nov 02 08:03:06.815387 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.814999    4898 scope.go:122] "RemoveContainer" containerID="6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3"
Nov 02 08:03:06.816591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.816564    4898 generic.go:356] "Generic (PLEG): container finished" podID="2950e4a5-896e-4388-ba7e-accccdf2153a" containerID="62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4" exitCode=0
Nov 02 08:03:06.816731 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.816603    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"
Nov 02 08:03:06.816731 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.816618    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl" event={"ID":"2950e4a5-896e-4388-ba7e-accccdf2153a","Type":"ContainerDied","Data":"62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4"}
Nov 02 08:03:06.816731 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.816661    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl" event={"ID":"2950e4a5-896e-4388-ba7e-accccdf2153a","Type":"ContainerDied","Data":"4e52bf90e29bfefd80f30726d5c5da1cf94f2e5de6e293d43808ab584a6235dc"}
Nov 02 08:03:06.838918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.838868    4898 scope.go:122] "RemoveContainer" containerID="6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3"
Nov 02 08:03:06.839242 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:03:06.839218    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3\": container with ID starting with 6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3 not found: ID does not exist" containerID="6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3"
Nov 02 08:03:06.839309 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.839254    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3"} err="failed to get container status \"6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3\": rpc error: code = NotFound desc = could not find container \"6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3\": container with ID starting with 6790c834b69d2ba6f177cc64f494b4aacc3a312e599c57ce25e6e26f02dd0ac3 not found: ID does not exist"
Nov 02 08:03:06.839309 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.839274    4898 scope.go:122] "RemoveContainer" containerID="62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4"
Nov 02 08:03:06.850434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.850409    4898 scope.go:122] "RemoveContainer" containerID="62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4"
Nov 02 08:03:06.850792 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:03:06.850761    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4\": container with ID starting with 62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4 not found: ID does not exist" containerID="62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4"
Nov 02 08:03:06.850868 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.850804    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4"} err="failed to get container status \"62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4\": rpc error: code = NotFound desc = could not find container \"62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4\": container with ID starting with 62a0d75326891bad7a5abd04ab965cf6ebc3f5d78b1bd1b34381459e8299a0d4 not found: ID does not exist"
Nov 02 08:03:06.889158 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.889121    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"]
Nov 02 08:03:06.898463 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.898432    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7a8bb6-7d87956497-89pr8"]
Nov 02 08:03:06.926590 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.926562    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"]
Nov 02 08:03:06.936451 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:06.936424    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7a8bb6-bb857998c-k7vwl"]
Nov 02 08:03:07.469421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:07.469383    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2950e4a5-896e-4388-ba7e-accccdf2153a" path="/var/lib/kubelet/pods/2950e4a5-896e-4388-ba7e-accccdf2153a/volumes"
Nov 02 08:03:07.469870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:07.469856    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e973ac4-18ee-48dd-8b9c-6d25ac250822" path="/var/lib/kubelet/pods/2e973ac4-18ee-48dd-8b9c-6d25ac250822/volumes"
Nov 02 08:03:12.414868 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:12.414831    4898 scope.go:122] "RemoveContainer" containerID="b8618d5f939b66fa9a2a091bb922ca499a1df4c8b141baa0d1eefb934fa01f3f"
Nov 02 08:03:12.425986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:12.425961    4898 scope.go:122] "RemoveContainer" containerID="35729fa9dfe327ec7989e8008e873b3a48ad3cd09925adfc5836d62a30be3654"
Nov 02 08:03:15.124574 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:03:15.124453    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:03:15.792031 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.791958    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"]
Nov 02 08:03:15.793336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.793306    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2950e4a5-896e-4388-ba7e-accccdf2153a" containerName="payload-pre-processing"
Nov 02 08:03:15.793516 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.793499    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2950e4a5-896e-4388-ba7e-accccdf2153a" containerName="payload-pre-processing"
Nov 02 08:03:15.793651 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.793639    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2e973ac4-18ee-48dd-8b9c-6d25ac250822" containerName="payload-processing"
Nov 02 08:03:15.793740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.793727    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e973ac4-18ee-48dd-8b9c-6d25ac250822" containerName="payload-processing"
Nov 02 08:03:15.794121 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.794102    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e973ac4-18ee-48dd-8b9c-6d25ac250822" containerName="payload-processing"
Nov 02 08:03:15.794263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.794251    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2950e4a5-896e-4388-ba7e-accccdf2153a" containerName="payload-pre-processing"
Nov 02 08:03:15.818955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.818917    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"]
Nov 02 08:03:15.819405 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.819318    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:15.870306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.870076    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"openshift-service-ca.crt\""
Nov 02 08:03:15.870306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.870141    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"agc-cred-74b36e68d5bd\""
Nov 02 08:03:15.870306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.870169    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"routing-overlay\""
Nov 02 08:03:15.870306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.870227    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-e2e--1a3816b7-dockercfg-4nccn\""
Nov 02 08:03:15.870306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.870280    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"kube-root-ca.crt\""
Nov 02 08:03:15.870306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.870285    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2dde84\""
Nov 02 08:03:15.873960 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.871540    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:15.873960 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.871726    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:15.873960 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.871811    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:15.873960 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.871948    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66lm\" (UniqueName: \"kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-kube-api-access-z66lm\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:15.972983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.972949    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66lm\" (UniqueName: \"kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-kube-api-access-z66lm\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:15.973244 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.973221    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:15.973399 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.973383    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:15.973485 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.973470    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:15.974152 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.974127    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:15.975284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.975254    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:15.976882 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.976849    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:16.002224 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:15.998326    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66lm\" (UniqueName: \"kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-kube-api-access-z66lm\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:16.178763 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:16.178718    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:16.546250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:16.546214    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"]
Nov 02 08:03:16.986787 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:16.986748    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg" event={"ID":"c86d659e-732e-42e3-bbcd-f92548e031b9","Type":"ContainerStarted","Data":"942f94cb2a2a573637976c3695e4d66fc3d79373d6390ba2b57a8a3ffef88251"}
Nov 02 08:03:16.987126 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:16.986812    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg" event={"ID":"c86d659e-732e-42e3-bbcd-f92548e031b9","Type":"ContainerStarted","Data":"a57b4284d5adb791baaa70569435675e4ea04bb8425c2805de0cf5e43976d6d2"}
Nov 02 08:03:16.987126 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:16.986843    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:17.051333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:17.051272    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg" podStartSLOduration=2.051251128 podStartE2EDuration="2.051251128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:03:17.045810815 +0000 UTC m=+2347.736137790" watchObservedRunningTime="2026-11-02 08:03:17.051251128 +0000 UTC m=+2347.741578088"
Nov 02 08:03:19.922263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:19.922230    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"]
Nov 02 08:03:19.923029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:19.922995    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg" podUID="4f9c024c-f87e-4232-8588-e48db8f85ba9" containerName="istio-proxy" containerID="cri-o://c688a8b311f44765014aea11de8e40fde3145fedf38ad339717294154599fd81" gracePeriod=30
Nov 02 08:03:20.015791 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:20.015758    4898 generic.go:356] "Generic (PLEG): container finished" podID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerID="5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963" exitCode=0
Nov 02 08:03:20.015991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:20.015833    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" event={"ID":"8d3b67b4-66e9-4f49-942d-61b4b632ac82","Type":"ContainerDied","Data":"5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963"}
Nov 02 08:03:20.016323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:20.016307    4898 scope.go:122] "RemoveContainer" containerID="5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963"
Nov 02 08:03:20.018090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:20.018060    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:03:20.018181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:20.018129    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:03:20.018181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:20.018170    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:03:21.026830 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:21.026796    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" event={"ID":"8d3b67b4-66e9-4f49-942d-61b4b632ac82","Type":"ContainerStarted","Data":"f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b"}
Nov 02 08:03:21.418863 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:21.418829    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:03:21.419272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:21.419249    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 08:03:21.419454 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:21.419424    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 08:03:22.246049 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.246015    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"]
Nov 02 08:03:22.249787 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.249757    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.309674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.309644    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"]
Nov 02 08:03:22.344575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.344534    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-envoy\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.344769 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.344597    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-token\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.344769 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.344701    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-certs\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.344769 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.344768    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-data\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.344982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.344792    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-socket\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.344982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.344931    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istiod-ca-cert\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.345092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.344982    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w674\" (UniqueName: \"kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-kube-api-access-2w674\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.345154 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.345134    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-podinfo\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.345213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.345172    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-credential-socket\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.349589 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.349559    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0f2e06-openshift-default-dockercfg-7bnpr\""
Nov 02 08:03:22.413402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.413365    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 08:03:22.413762 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.413732    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 08:03:22.453949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.446739    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-data\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.468954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.454213    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-socket\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.469360 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.469328    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istiod-ca-cert\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.469553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.469534    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w674\" (UniqueName: \"kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-kube-api-access-2w674\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.469760 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.469740    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-podinfo\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.469943 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.469924    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-credential-socket\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.470094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.470077    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-envoy\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.470268 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.470243    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-token\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.470473 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.470452    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-certs\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.471114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.471087    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-certs\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.475261 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.460363    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-socket\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.500025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.477342    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-credential-socket\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.500258 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.491118    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istiod-ca-cert\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.500384 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.460499    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-data\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.518909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.516859    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w674\" (UniqueName: \"kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-kube-api-access-2w674\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.538472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.535637    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-podinfo\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.538472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.537452    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-token\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.555911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.544163    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-envoy\") pod \"e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") " pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:22.657467 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:22.657431    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:23.106782 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:23.106745    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"]
Nov 02 08:03:23.108791 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:23.108757    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:03:23.108968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:23.108951    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:03:23.109075 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:23.109063    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:03:23.413430 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:23.413395    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 08:03:23.413950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:23.413454    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 08:03:24.094637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:24.094592    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" event={"ID":"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d","Type":"ContainerStarted","Data":"7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d"}
Nov 02 08:03:24.094829 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:24.094651    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" event={"ID":"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d","Type":"ContainerStarted","Data":"5265d338182e784caadfa77c22c734c2a3bde491e42bd6ea7f61de7cda9b3cf3"}
Nov 02 08:03:24.144824 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:24.144770    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" podStartSLOduration=2.144755563 podStartE2EDuration="2.144755563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:03:24.139641469 +0000 UTC m=+2354.829968463" watchObservedRunningTime="2026-11-02 08:03:24.144755563 +0000 UTC m=+2354.835082527"
Nov 02 08:03:24.413116 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:24.413078    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 08:03:24.413310 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:24.413136    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 08:03:24.672273 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:24.672188    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:24.692760 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:24.692729    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:25.108933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.108860    4898 generic.go:356] "Generic (PLEG): container finished" podID="4f9c024c-f87e-4232-8588-e48db8f85ba9" containerID="c688a8b311f44765014aea11de8e40fde3145fedf38ad339717294154599fd81" exitCode=0
Nov 02 08:03:25.110051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.110007    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg" event={"ID":"4f9c024c-f87e-4232-8588-e48db8f85ba9","Type":"ContainerDied","Data":"c688a8b311f44765014aea11de8e40fde3145fedf38ad339717294154599fd81"}
Nov 02 08:03:25.110214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.110088    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:25.110897 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.110866    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:03:25.335773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.335745    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:03:25.412692 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.412657    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 08:03:25.412924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.412709    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 08:03:25.511002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.510966    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-certs\") pod \"4f9c024c-f87e-4232-8588-e48db8f85ba9\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") "
Nov 02 08:03:25.511172 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.511041    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-socket\") pod \"4f9c024c-f87e-4232-8588-e48db8f85ba9\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") "
Nov 02 08:03:25.511172 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.511115    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-envoy\") pod \"4f9c024c-f87e-4232-8588-e48db8f85ba9\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") "
Nov 02 08:03:25.511172 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.511140    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-token\" (UniqueName: \"kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-token\") pod \"4f9c024c-f87e-4232-8588-e48db8f85ba9\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") "
Nov 02 08:03:25.511427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.511226    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f9c024c-f87e-4232-8588-e48db8f85ba9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9c024c-f87e-4232-8588-e48db8f85ba9-istiod-ca-cert\") pod \"4f9c024c-f87e-4232-8588-e48db8f85ba9\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") "
Nov 02 08:03:25.511427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.511329    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-data\") pod \"4f9c024c-f87e-4232-8588-e48db8f85ba9\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") "
Nov 02 08:03:25.511427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.511349    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-podinfo\") pod \"4f9c024c-f87e-4232-8588-e48db8f85ba9\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") "
Nov 02 08:03:25.511427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.511395    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-credential-socket\") pod \"4f9c024c-f87e-4232-8588-e48db8f85ba9\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") "
Nov 02 08:03:25.511644 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.511451    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-kube-api-access-4zvmx\" (UniqueName: \"kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-kube-api-access-4zvmx\") pod \"4f9c024c-f87e-4232-8588-e48db8f85ba9\" (UID: \"4f9c024c-f87e-4232-8588-e48db8f85ba9\") "
Nov 02 08:03:25.512964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.512936    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-certs" pod "4f9c024c-f87e-4232-8588-e48db8f85ba9" (UID: "4f9c024c-f87e-4232-8588-e48db8f85ba9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:03:25.513131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.512951    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-data" pod "4f9c024c-f87e-4232-8588-e48db8f85ba9" (UID: "4f9c024c-f87e-4232-8588-e48db8f85ba9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:03:25.513207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.512981    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9c024c-f87e-4232-8588-e48db8f85ba9-istiod-ca-cert" pod "4f9c024c-f87e-4232-8588-e48db8f85ba9" (UID: "4f9c024c-f87e-4232-8588-e48db8f85ba9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:03:25.513276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.512990    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-socket" pod "4f9c024c-f87e-4232-8588-e48db8f85ba9" (UID: "4f9c024c-f87e-4232-8588-e48db8f85ba9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:03:25.513570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.513543    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-credential-socket" pod "4f9c024c-f87e-4232-8588-e48db8f85ba9" (UID: "4f9c024c-f87e-4232-8588-e48db8f85ba9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:03:25.515586 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.515556    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-kube-api-access-4zvmx" pod "4f9c024c-f87e-4232-8588-e48db8f85ba9" (UID: "4f9c024c-f87e-4232-8588-e48db8f85ba9"). InnerVolumeSpecName "kube-api-access-4zvmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:03:25.516001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.515975    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-podinfo" pod "4f9c024c-f87e-4232-8588-e48db8f85ba9" (UID: "4f9c024c-f87e-4232-8588-e48db8f85ba9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:03:25.516383 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.516363    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-token" pod "4f9c024c-f87e-4232-8588-e48db8f85ba9" (UID: "4f9c024c-f87e-4232-8588-e48db8f85ba9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:03:25.516807 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.516787    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-envoy" pod "4f9c024c-f87e-4232-8588-e48db8f85ba9" (UID: "4f9c024c-f87e-4232-8588-e48db8f85ba9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:03:25.613817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.613724    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zvmx\" (UniqueName: \"kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-kube-api-access-4zvmx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.613817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.613755    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.613817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.613767    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.613817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.613779    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.613817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.613790    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.613817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.613798    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9c024c-f87e-4232-8588-e48db8f85ba9-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.613817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.613805    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.613817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.613813    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9c024c-f87e-4232-8588-e48db8f85ba9-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:25.613817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:25.613821    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9c024c-f87e-4232-8588-e48db8f85ba9-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:26.121148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:26.121115    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"
Nov 02 08:03:26.121632 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:26.121124    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg" event={"ID":"4f9c024c-f87e-4232-8588-e48db8f85ba9","Type":"ContainerDied","Data":"e798113d01978eba4f6bc7574843bc2d05fca14677d480e9f28d1a500081e49f"}
Nov 02 08:03:26.121632 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:26.121255    4898 scope.go:122] "RemoveContainer" containerID="c688a8b311f44765014aea11de8e40fde3145fedf38ad339717294154599fd81"
Nov 02 08:03:26.197323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:26.197280    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"]
Nov 02 08:03:26.204951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:26.204915    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-7a8bb6-openshift-default-7f996bcbf5-bcbmg"]
Nov 02 08:03:26.412634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:26.412602    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 08:03:26.412823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:26.412652    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 08:03:27.418318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:27.418286    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:03:27.418799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:27.418779    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:03:27.419525 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:27.419504    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:03:27.466075 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:27.466031    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9c024c-f87e-4232-8588-e48db8f85ba9" path="/var/lib/kubelet/pods/4f9c024c-f87e-4232-8588-e48db8f85ba9/volumes"
Nov 02 08:03:28.002547 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.002516    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:03:28.372582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.372500    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"]
Nov 02 08:03:28.373264 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.373247    4898 cpu_manager.go:414] "containerMap: removing container" podUID="4f9c024c-f87e-4232-8588-e48db8f85ba9" containerName="istio-proxy"
Nov 02 08:03:28.373318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.373267    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9c024c-f87e-4232-8588-e48db8f85ba9" containerName="istio-proxy"
Nov 02 08:03:28.373381 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.373371    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9c024c-f87e-4232-8588-e48db8f85ba9" containerName="istio-proxy"
Nov 02 08:03:28.377097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.377061    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"
Nov 02 08:03:28.393701 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.393667    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0f2e06\""
Nov 02 08:03:28.397281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.397252    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"]
Nov 02 08:03:28.440564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.440531    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"]
Nov 02 08:03:28.444071 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.444052    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"
Nov 02 08:03:28.462984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.462946    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0f2e06-dockercfg-k4n2f\""
Nov 02 08:03:28.463196 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.463178    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"]
Nov 02 08:03:28.621865 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.621811    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c36a913f-4117-4638-881a-c15c2938f172-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4\" (UID: \"c36a913f-4117-4638-881a-c15c2938f172\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"
Nov 02 08:03:28.622124 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.621916    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec39c96-e0ca-439f-90be-331900d88b53-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq\" (UID: \"eec39c96-e0ca-439f-90be-331900d88b53\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"
Nov 02 08:03:28.622124 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.622063    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv9c6\" (UniqueName: \"kubernetes.io/projected/eec39c96-e0ca-439f-90be-331900d88b53-kube-api-access-xv9c6\") pod \"payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq\" (UID: \"eec39c96-e0ca-439f-90be-331900d88b53\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"
Nov 02 08:03:28.622124 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.622113    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkql\" (UniqueName: \"kubernetes.io/projected/c36a913f-4117-4638-881a-c15c2938f172-kube-api-access-gzkql\") pod \"payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4\" (UID: \"c36a913f-4117-4638-881a-c15c2938f172\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"
Nov 02 08:03:28.724839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.724783    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c36a913f-4117-4638-881a-c15c2938f172-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4\" (UID: \"c36a913f-4117-4638-881a-c15c2938f172\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"
Nov 02 08:03:28.725113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.724854    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec39c96-e0ca-439f-90be-331900d88b53-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq\" (UID: \"eec39c96-e0ca-439f-90be-331900d88b53\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"
Nov 02 08:03:28.725113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.724923    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv9c6\" (UniqueName: \"kubernetes.io/projected/eec39c96-e0ca-439f-90be-331900d88b53-kube-api-access-xv9c6\") pod \"payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq\" (UID: \"eec39c96-e0ca-439f-90be-331900d88b53\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"
Nov 02 08:03:28.725113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.724968    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkql\" (UniqueName: \"kubernetes.io/projected/c36a913f-4117-4638-881a-c15c2938f172-kube-api-access-gzkql\") pod \"payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4\" (UID: \"c36a913f-4117-4638-881a-c15c2938f172\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"
Nov 02 08:03:28.726095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.726067    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c36a913f-4117-4638-881a-c15c2938f172-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4\" (UID: \"c36a913f-4117-4638-881a-c15c2938f172\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"
Nov 02 08:03:28.726998 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.726969    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec39c96-e0ca-439f-90be-331900d88b53-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq\" (UID: \"eec39c96-e0ca-439f-90be-331900d88b53\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"
Nov 02 08:03:28.749136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.749104    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkql\" (UniqueName: \"kubernetes.io/projected/c36a913f-4117-4638-881a-c15c2938f172-kube-api-access-gzkql\") pod \"payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4\" (UID: \"c36a913f-4117-4638-881a-c15c2938f172\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"
Nov 02 08:03:28.749136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.749128    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv9c6\" (UniqueName: \"kubernetes.io/projected/eec39c96-e0ca-439f-90be-331900d88b53-kube-api-access-xv9c6\") pod \"payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq\" (UID: \"eec39c96-e0ca-439f-90be-331900d88b53\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"
Nov 02 08:03:28.758813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.758773    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"
Nov 02 08:03:28.992549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:28.992443    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"
Nov 02 08:03:29.096630 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:03:29.096581    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36a913f_4117_4638_881a_c15c2938f172.slice/crio-5e21254c498c0bf5fce8cfac966fa97e3f0c45bd67680363b32439ec259bec9d WatchSource:0}: Error finding container 5e21254c498c0bf5fce8cfac966fa97e3f0c45bd67680363b32439ec259bec9d: Status 404 returned error can't find the container with id 5e21254c498c0bf5fce8cfac966fa97e3f0c45bd67680363b32439ec259bec9d
Nov 02 08:03:29.099665 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:29.099132    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"]
Nov 02 08:03:29.151549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:29.151511    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4" event={"ID":"c36a913f-4117-4638-881a-c15c2938f172","Type":"ContainerStarted","Data":"5e21254c498c0bf5fce8cfac966fa97e3f0c45bd67680363b32439ec259bec9d"}
Nov 02 08:03:29.298137 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:29.298109    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"]
Nov 02 08:03:29.299330 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:03:29.299300    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec39c96_e0ca_439f_90be_331900d88b53.slice/crio-0e4fe5acf14684d63ea536e36fd31e30a7bcddc57f524dbc9d9369ce199224c8 WatchSource:0}: Error finding container 0e4fe5acf14684d63ea536e36fd31e30a7bcddc57f524dbc9d9369ce199224c8: Status 404 returned error can't find the container with id 0e4fe5acf14684d63ea536e36fd31e30a7bcddc57f524dbc9d9369ce199224c8
Nov 02 08:03:30.161650 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:30.161611    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4" event={"ID":"c36a913f-4117-4638-881a-c15c2938f172","Type":"ContainerStarted","Data":"4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20"}
Nov 02 08:03:30.162162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:30.161703    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"
Nov 02 08:03:30.163550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:30.163520    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq" event={"ID":"eec39c96-e0ca-439f-90be-331900d88b53","Type":"ContainerStarted","Data":"e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32"}
Nov 02 08:03:30.163663 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:30.163565    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq" event={"ID":"eec39c96-e0ca-439f-90be-331900d88b53","Type":"ContainerStarted","Data":"0e4fe5acf14684d63ea536e36fd31e30a7bcddc57f524dbc9d9369ce199224c8"}
Nov 02 08:03:30.163663 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:30.163586    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"
Nov 02 08:03:30.204822 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:30.204763    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4" podStartSLOduration=2.204750208 podStartE2EDuration="2.204750208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:03:30.20208568 +0000 UTC m=+2360.892412645" watchObservedRunningTime="2026-11-02 08:03:30.204750208 +0000 UTC m=+2360.895077170"
Nov 02 08:03:30.246174 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:30.246122    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq" podStartSLOduration=2.246108712 podStartE2EDuration="2.246108712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:03:30.24281146 +0000 UTC m=+2360.933138426" watchObservedRunningTime="2026-11-02 08:03:30.246108712 +0000 UTC m=+2360.936435700"
Nov 02 08:03:31.033275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:31.033238    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"]
Nov 02 08:03:31.045481 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:31.045448    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ff88e3-6x24f"]
Nov 02 08:03:31.466412 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:31.466376    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab85ca9-578d-4911-ab8f-89f8505ddd40" path="/var/lib/kubelet/pods/1ab85ca9-578d-4911-ab8f-89f8505ddd40/volumes"
Nov 02 08:03:36.110159 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:36.110121    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"]
Nov 02 08:03:41.034069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.034034    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"]
Nov 02 08:03:41.045619 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.045588    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41e3a623-bzg7t"]
Nov 02 08:03:41.110419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.110375    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t" podUID="e0b81fa1-5a4b-439b-baae-d654f7c0f373" containerName="payload-processing" containerID="cri-o://f0d9f7cf6f584fb85ff1f1150ee4d978c95a2bb2fdd43cd9a5f835c3358bb7fa" gracePeriod=25
Nov 02 08:03:41.174934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.174869    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"
Nov 02 08:03:41.175085 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.175023    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"
Nov 02 08:03:41.300934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.299562    4898 generic.go:356] "Generic (PLEG): container finished" podID="e0b81fa1-5a4b-439b-baae-d654f7c0f373" containerID="f0d9f7cf6f584fb85ff1f1150ee4d978c95a2bb2fdd43cd9a5f835c3358bb7fa" exitCode=0
Nov 02 08:03:41.301735 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.301705    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t" event={"ID":"e0b81fa1-5a4b-439b-baae-d654f7c0f373","Type":"ContainerDied","Data":"f0d9f7cf6f584fb85ff1f1150ee4d978c95a2bb2fdd43cd9a5f835c3358bb7fa"}
Nov 02 08:03:41.468332 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.468272    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76ec918-191f-46b5-ac33-98189801673a" path="/var/lib/kubelet/pods/d76ec918-191f-46b5-ac33-98189801673a/volumes"
Nov 02 08:03:41.494946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.494911    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:03:41.523338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.523309    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-routing-overlay-volume\") pod \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") "
Nov 02 08:03:41.523522 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.523500    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-kube-api-access-xv6bw\" (UniqueName: \"kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-kube-api-access-xv6bw\") pod \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") "
Nov 02 08:03:41.523597 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.523570    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-provider-credentials\") pod \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") "
Nov 02 08:03:41.523637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.523602    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-plugins-config-volume\") pod \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\" (UID: \"e0b81fa1-5a4b-439b-baae-d654f7c0f373\") "
Nov 02 08:03:41.523674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.523645    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-routing-overlay-volume" pod "e0b81fa1-5a4b-439b-baae-d654f7c0f373" (UID: "e0b81fa1-5a4b-439b-baae-d654f7c0f373"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:03:41.524152 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.524119    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:41.524152 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.524125    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-plugins-config-volume" pod "e0b81fa1-5a4b-439b-baae-d654f7c0f373" (UID: "e0b81fa1-5a4b-439b-baae-d654f7c0f373"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:03:41.525934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.525908    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-provider-credentials" pod "e0b81fa1-5a4b-439b-baae-d654f7c0f373" (UID: "e0b81fa1-5a4b-439b-baae-d654f7c0f373"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:03:41.526037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.526021    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-kube-api-access-xv6bw" pod "e0b81fa1-5a4b-439b-baae-d654f7c0f373" (UID: "e0b81fa1-5a4b-439b-baae-d654f7c0f373"). InnerVolumeSpecName "kube-api-access-xv6bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:03:41.625525 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.625444    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv6bw\" (UniqueName: \"kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-kube-api-access-xv6bw\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:41.625525 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.625475    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0b81fa1-5a4b-439b-baae-d654f7c0f373-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:41.625525 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:41.625484    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b81fa1-5a4b-439b-baae-d654f7c0f373-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:03:42.311504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:42.311475    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"
Nov 02 08:03:42.311504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:42.311488    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t" event={"ID":"e0b81fa1-5a4b-439b-baae-d654f7c0f373","Type":"ContainerDied","Data":"6e5f71c0733b9b6db2ac2dd3d0d270e2048e487bb91fe8e5e9bc00d6aec6a8c8"}
Nov 02 08:03:42.312094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:42.311544    4898 scope.go:122] "RemoveContainer" containerID="f0d9f7cf6f584fb85ff1f1150ee4d978c95a2bb2fdd43cd9a5f835c3358bb7fa"
Nov 02 08:03:42.378959 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:42.378929    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"]
Nov 02 08:03:42.389240 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:42.389216    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b948kgp8t"]
Nov 02 08:03:43.512599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:43.512559    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b81fa1-5a4b-439b-baae-d654f7c0f373" path="/var/lib/kubelet/pods/e0b81fa1-5a4b-439b-baae-d654f7c0f373/volumes"
Nov 02 08:03:45.051074 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:03:45.051036    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:03:50.970136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:50.970100    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"]
Nov 02 08:03:50.970881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:50.970858    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e0b81fa1-5a4b-439b-baae-d654f7c0f373" containerName="payload-processing"
Nov 02 08:03:50.970881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:50.970879    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b81fa1-5a4b-439b-baae-d654f7c0f373" containerName="payload-processing"
Nov 02 08:03:50.971132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:50.971116    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b81fa1-5a4b-439b-baae-d654f7c0f373" containerName="payload-processing"
Nov 02 08:03:50.977245 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:50.977220    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:50.997508 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:50.997473    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"]
Nov 02 08:03:50.998137 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:50.998111    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"routing-overlay\""
Nov 02 08:03:51.003514 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.003489    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-plugins-e2e-extmod-b-643355\""
Nov 02 08:03:51.004061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.004041    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"openshift-service-ca.crt\""
Nov 02 08:03:51.004288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.004273    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-cce518e50d6f\""
Nov 02 08:03:51.004434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.004419    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-34c73b51c090\""
Nov 02 08:03:51.004680 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.004663    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"kube-root-ca.crt\""
Nov 02 08:03:51.004878 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.004863    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-bfdb1bbb5e7a\""
Nov 02 08:03:51.005193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.005172    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"payload-processing-external-model-e2e--9fb8cda0-dockercfg-h9ktn\""
Nov 02 08:03:51.100933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.100849    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-86967dxxq\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.101146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.101086    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-86967dxxq\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.101225 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.101209    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzzrt\" (UniqueName: \"kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-kube-api-access-wzzrt\") pod \"payload-processing-external-model-e2e-extmod-b-643355-86967dxxq\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.101292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.101270    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-86967dxxq\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.203503 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.203453    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-86967dxxq\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.203685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.203582    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzzrt\" (UniqueName: \"kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-kube-api-access-wzzrt\") pod \"payload-processing-external-model-e2e-extmod-b-643355-86967dxxq\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.203685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.203640    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-86967dxxq\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.203805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.203736    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-86967dxxq\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.204308 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.204278    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-86967dxxq\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.204459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.204350    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-86967dxxq\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.206358 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.206335    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-86967dxxq\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.227549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.227486    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzzrt\" (UniqueName: \"kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-kube-api-access-wzzrt\") pod \"payload-processing-external-model-e2e-extmod-b-643355-86967dxxq\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.294714 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.294667    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:51.603793 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:03:51.603754    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c5cea7_0dcd_4eea_ab45_d80f656f0861.slice/crio-cd6bcd9fa5b236e2cb22df0c9df213c122b0757432e8b86af2cf674a22db4037 WatchSource:0}: Error finding container cd6bcd9fa5b236e2cb22df0c9df213c122b0757432e8b86af2cf674a22db4037: Status 404 returned error can't find the container with id cd6bcd9fa5b236e2cb22df0c9df213c122b0757432e8b86af2cf674a22db4037
Nov 02 08:03:51.604226 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.604202    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"]
Nov 02 08:03:51.867346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.867232    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"]
Nov 02 08:03:51.872097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.872073    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:51.892645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.892618    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"kube-root-ca.crt\""
Nov 02 08:03:51.892967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.892850    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bb4907\""
Nov 02 08:03:51.898631 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.898606    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"]
Nov 02 08:03:51.906113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.906087    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"agc-cred-1d1f0c1265f1\""
Nov 02 08:03:51.906319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.906086    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"agc-cred-2a9bfe236817\""
Nov 02 08:03:51.906319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.906124    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"routing-overlay\""
Nov 02 08:03:51.906474 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.906124    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"payload-processing-external-model-e2e--634ed34c-dockercfg-lmlct\""
Nov 02 08:03:51.906474 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.906129    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"openshift-service-ca.crt\""
Nov 02 08:03:51.906474 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:51.906191    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bb4907\"/\"agc-cred-f32e6bb1571b\""
Nov 02 08:03:52.015192 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.015153    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbqrl\" (UniqueName: \"kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-kube-api-access-kbqrl\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.015638 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.015215    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.015638 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.015321    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.015638 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.015469    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.121076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.120094    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbqrl\" (UniqueName: \"kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-kube-api-access-kbqrl\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.121076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.120200    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.121076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.120253    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.121076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.120356    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.130912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.121871    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.130912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.128429    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.131874 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.131840    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.173617 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.173581    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbqrl\" (UniqueName: \"kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-kube-api-access-kbqrl\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.189204 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.189166    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:52.446231 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.446190    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq" event={"ID":"71c5cea7-0dcd-4eea-ab45-d80f656f0861","Type":"ContainerStarted","Data":"26229c12666dc74b4350165d062253eaacb34d4724016dbdffc6ffeb8b15a203"}
Nov 02 08:03:52.446455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.446253    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq" event={"ID":"71c5cea7-0dcd-4eea-ab45-d80f656f0861","Type":"ContainerStarted","Data":"cd6bcd9fa5b236e2cb22df0c9df213c122b0757432e8b86af2cf674a22db4037"}
Nov 02 08:03:52.446569 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.446551    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:03:52.565426 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.565361    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq" podStartSLOduration=2.565341651 podStartE2EDuration="2.565341651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:03:52.491215034 +0000 UTC m=+2383.181542024" watchObservedRunningTime="2026-11-02 08:03:52.565341651 +0000 UTC m=+2383.255668615"
Nov 02 08:03:52.570945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:52.570912    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"]
Nov 02 08:03:52.571397 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:03:52.571372    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7eb816c_6af2_472e_b255_8f5cc7f1099d.slice/crio-7dd8bb9f113816d4f904263cf54b2c5e8aa51b3065792ab3847b7a07d92ee46d WatchSource:0}: Error finding container 7dd8bb9f113816d4f904263cf54b2c5e8aa51b3065792ab3847b7a07d92ee46d: Status 404 returned error can't find the container with id 7dd8bb9f113816d4f904263cf54b2c5e8aa51b3065792ab3847b7a07d92ee46d
Nov 02 08:03:53.455694 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:53.455652    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x" event={"ID":"a7eb816c-6af2-472e-b255-8f5cc7f1099d","Type":"ContainerStarted","Data":"7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53"}
Nov 02 08:03:53.456242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:53.455710    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x" event={"ID":"a7eb816c-6af2-472e-b255-8f5cc7f1099d","Type":"ContainerStarted","Data":"7dd8bb9f113816d4f904263cf54b2c5e8aa51b3065792ab3847b7a07d92ee46d"}
Nov 02 08:03:53.456242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:53.455736    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:03:53.500267 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:03:53.500217    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x" podStartSLOduration=2.500189833 podStartE2EDuration="2.500189833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:03:53.500088173 +0000 UTC m=+2384.190415138" watchObservedRunningTime="2026-11-02 08:03:53.500189833 +0000 UTC m=+2384.190516796"
Nov 02 08:04:00.070927 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:00.068947    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"]
Nov 02 08:04:00.100916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:00.098779    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-395b6f0b-p7ftm"]
Nov 02 08:04:01.467718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:01.467671    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007bb236-057e-450e-85ef-a49291f5a91c" path="/var/lib/kubelet/pods/007bb236-057e-450e-85ef-a49291f5a91c/volumes"
Nov 02 08:04:03.467160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:03.467130    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:04:04.469539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:04.469507    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:04:10.033091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:10.033054    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"]
Nov 02 08:04:10.047937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:10.047883    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fbb756d-jgdnk"]
Nov 02 08:04:10.318658 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:10.318559    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:04:10.325339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:10.325310    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:04:11.467563 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:11.467518    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4517047f-61ba-4cf9-a710-36e767dc7038" path="/var/lib/kubelet/pods/4517047f-61ba-4cf9-a710-36e767dc7038/volumes"
Nov 02 08:04:14.998475 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:04:14.998435    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:04:33.964794 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:33.964759    4898 generic.go:356] "Generic (PLEG): container finished" podID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerID="7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d" exitCode=0
Nov 02 08:04:33.965361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:33.964837    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" event={"ID":"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d","Type":"ContainerDied","Data":"7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d"}
Nov 02 08:04:33.965361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:33.965327    4898 scope.go:122] "RemoveContainer" containerID="7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d"
Nov 02 08:04:33.967287 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:33.967254    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:04:33.967374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:33.967327    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:04:33.967374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:33.967363    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:04:34.993278 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:34.993236    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" event={"ID":"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d","Type":"ContainerStarted","Data":"24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f"}
Nov 02 08:04:35.663222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:35.663189    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:04:35.670022 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:35.669987    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Nov 02 08:04:35.670279 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:35.670253    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Nov 02 08:04:36.663913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:36.661248    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Nov 02 08:04:36.663913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:36.661310    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Nov 02 08:04:37.667217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:37.667178    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Nov 02 08:04:37.668530 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:37.668497    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Nov 02 08:04:38.669017 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:38.668976    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Nov 02 08:04:38.669646 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:38.669610    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Nov 02 08:04:39.034951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:39.034857    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"]
Nov 02 08:04:39.044686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:39.044656    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b73536d-6t8tt"]
Nov 02 08:04:39.490871 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:39.490824    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2f0387-92d1-464d-b3be-eb584e706018" path="/var/lib/kubelet/pods/4c2f0387-92d1-464d-b3be-eb584e706018/volumes"
Nov 02 08:04:39.675540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:39.675499    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body=
Nov 02 08:04:39.676152 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:39.676117    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused"
Nov 02 08:04:40.038150 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:40.038114    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"]
Nov 02 08:04:40.049537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:40.049497    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7899d9f9-dw996"]
Nov 02 08:04:40.110926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:40.110876    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:04:40.665743 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:40.665705    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:04:41.046085 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:41.045998    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:04:41.467119 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:41.467083    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea00cdda-b2c6-4e94-bb14-12c924b61b79" path="/var/lib/kubelet/pods/ea00cdda-b2c6-4e94-bb14-12c924b61b79/volumes"
Nov 02 08:04:44.990012 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:04:44.989956    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:04:50.032543 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:50.032504    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"]
Nov 02 08:04:50.044821 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:50.044790    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44e0bb5d-fndhj"]
Nov 02 08:04:51.038602 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:51.038563    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"]
Nov 02 08:04:51.050289 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:51.050259    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a310ec-nzvst"]
Nov 02 08:04:51.468511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:51.468464    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b71e78-38e8-491a-8806-88d576ab0696" path="/var/lib/kubelet/pods/69b71e78-38e8-491a-8806-88d576ab0696/volumes"
Nov 02 08:04:51.469047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:51.469028    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8147482-0f1f-4dbe-bf62-5886c237bfe4" path="/var/lib/kubelet/pods/d8147482-0f1f-4dbe-bf62-5886c237bfe4/volumes"
Nov 02 08:04:58.131993 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.131959    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"]
Nov 02 08:04:58.138005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.137985    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.156549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.156522    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"]
Nov 02 08:04:58.206737 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.206696    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.207004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.206751    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.207098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.207018    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ss97\" (UniqueName: \"kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-kube-api-access-4ss97\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.207098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.207088    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.308067 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.308024    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ss97\" (UniqueName: \"kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-kube-api-access-4ss97\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.308299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.308110    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.308299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.308177    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.308299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.308210    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.312620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.309287    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.312620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.309640    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.313523 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.313490    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.332966 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.332936    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ss97\" (UniqueName: \"kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-kube-api-access-4ss97\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.453840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.453809    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:58.894930 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:58.890839    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"]
Nov 02 08:04:59.332874 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:59.332835    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq" event={"ID":"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b","Type":"ContainerStarted","Data":"4ec46ef1640270f743a7243ef5ccd3bf3edd373a73ff6f7b55df81f423cbcdc7"}
Nov 02 08:04:59.333333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:59.332939    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq" event={"ID":"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b","Type":"ContainerStarted","Data":"79846ced2fb1223d27e72556fbafc82bd9e9700cadc26eef0c0d259525106c61"}
Nov 02 08:04:59.333333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:59.332980    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:04:59.378372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:04:59.378314    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq" podStartSLOduration=1.378296094 podStartE2EDuration="1.378296094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:04:59.375014929 +0000 UTC m=+2450.065341893" watchObservedRunningTime="2026-11-02 08:04:59.378296094 +0000 UTC m=+2450.068623061"
Nov 02 08:05:01.331801 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.331764    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"]
Nov 02 08:05:01.336559 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.336535    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"
Nov 02 08:05:01.356976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.356936    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"]
Nov 02 08:05:01.362407 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.362378    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2365fa-dockercfg-x686m\""
Nov 02 08:05:01.362675 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.362432    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2365fa\""
Nov 02 08:05:01.441167 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.430500    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"]
Nov 02 08:05:01.512960 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.460346    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac230396-f0f0-418f-bba6-81b0d8d28026-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk\" (UID: \"ac230396-f0f0-418f-bba6-81b0d8d28026\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"
Nov 02 08:05:01.515539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.515494    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxph5\" (UniqueName: \"kubernetes.io/projected/ac230396-f0f0-418f-bba6-81b0d8d28026-kube-api-access-mxph5\") pod \"payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk\" (UID: \"ac230396-f0f0-418f-bba6-81b0d8d28026\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"
Nov 02 08:05:01.590094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.590010    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"
Nov 02 08:05:01.617912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.617861    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxph5\" (UniqueName: \"kubernetes.io/projected/ac230396-f0f0-418f-bba6-81b0d8d28026-kube-api-access-mxph5\") pod \"payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk\" (UID: \"ac230396-f0f0-418f-bba6-81b0d8d28026\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"
Nov 02 08:05:01.618209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.618187    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac230396-f0f0-418f-bba6-81b0d8d28026-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk\" (UID: \"ac230396-f0f0-418f-bba6-81b0d8d28026\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"
Nov 02 08:05:01.619424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.619396    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac230396-f0f0-418f-bba6-81b0d8d28026-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk\" (UID: \"ac230396-f0f0-418f-bba6-81b0d8d28026\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"
Nov 02 08:05:01.635837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.631597    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"]
Nov 02 08:05:01.655480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.655444    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxph5\" (UniqueName: \"kubernetes.io/projected/ac230396-f0f0-418f-bba6-81b0d8d28026-kube-api-access-mxph5\") pod \"payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk\" (UID: \"ac230396-f0f0-418f-bba6-81b0d8d28026\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"
Nov 02 08:05:01.683454 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.683199    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"
Nov 02 08:05:01.720884 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.720842    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk6c6\" (UniqueName: \"kubernetes.io/projected/5f970daf-e1a2-4859-8446-328a1d214d2e-kube-api-access-kk6c6\") pod \"payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8\" (UID: \"5f970daf-e1a2-4859-8446-328a1d214d2e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"
Nov 02 08:05:01.722933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.722563    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f970daf-e1a2-4859-8446-328a1d214d2e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8\" (UID: \"5f970daf-e1a2-4859-8446-328a1d214d2e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"
Nov 02 08:05:01.824914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.823875    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk6c6\" (UniqueName: \"kubernetes.io/projected/5f970daf-e1a2-4859-8446-328a1d214d2e-kube-api-access-kk6c6\") pod \"payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8\" (UID: \"5f970daf-e1a2-4859-8446-328a1d214d2e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"
Nov 02 08:05:01.824914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.824030    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f970daf-e1a2-4859-8446-328a1d214d2e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8\" (UID: \"5f970daf-e1a2-4859-8446-328a1d214d2e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"
Nov 02 08:05:01.825206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.825144    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f970daf-e1a2-4859-8446-328a1d214d2e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8\" (UID: \"5f970daf-e1a2-4859-8446-328a1d214d2e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"
Nov 02 08:05:01.851017 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.850942    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk6c6\" (UniqueName: \"kubernetes.io/projected/5f970daf-e1a2-4859-8446-328a1d214d2e-kube-api-access-kk6c6\") pod \"payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8\" (UID: \"5f970daf-e1a2-4859-8446-328a1d214d2e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"
Nov 02 08:05:01.978296 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:01.978261    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"
Nov 02 08:05:02.044192 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:02.041301    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"]
Nov 02 08:05:02.046807 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:05:02.046779    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac230396_f0f0_418f_bba6_81b0d8d28026.slice/crio-898434b7399216e6097a53bab7f54ca07c8e227ffd8bbd950cb6c1313e101b80 WatchSource:0}: Error finding container 898434b7399216e6097a53bab7f54ca07c8e227ffd8bbd950cb6c1313e101b80: Status 404 returned error can't find the container with id 898434b7399216e6097a53bab7f54ca07c8e227ffd8bbd950cb6c1313e101b80
Nov 02 08:05:02.292755 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:05:02.292718    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f970daf_e1a2_4859_8446_328a1d214d2e.slice/crio-1292c8c89a1bfd218927e766acd81ba1411fd37eb2fcbc865a7c30351665ccbd WatchSource:0}: Error finding container 1292c8c89a1bfd218927e766acd81ba1411fd37eb2fcbc865a7c30351665ccbd: Status 404 returned error can't find the container with id 1292c8c89a1bfd218927e766acd81ba1411fd37eb2fcbc865a7c30351665ccbd
Nov 02 08:05:02.293437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:02.293408    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"]
Nov 02 08:05:02.362143 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:02.362109    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk" event={"ID":"ac230396-f0f0-418f-bba6-81b0d8d28026","Type":"ContainerStarted","Data":"f40c30546ccfc7aabaab1ed41881442ba275713af9e294ffd78e8238dcf76da1"}
Nov 02 08:05:02.362527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:02.362179    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk" event={"ID":"ac230396-f0f0-418f-bba6-81b0d8d28026","Type":"ContainerStarted","Data":"898434b7399216e6097a53bab7f54ca07c8e227ffd8bbd950cb6c1313e101b80"}
Nov 02 08:05:02.362527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:02.362209    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"
Nov 02 08:05:02.363708 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:02.363681    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8" event={"ID":"5f970daf-e1a2-4859-8446-328a1d214d2e","Type":"ContainerStarted","Data":"1292c8c89a1bfd218927e766acd81ba1411fd37eb2fcbc865a7c30351665ccbd"}
Nov 02 08:05:02.405492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:02.405442    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk" podStartSLOduration=1.405427119 podStartE2EDuration="1.405427119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:02.402377416 +0000 UTC m=+2453.092704398" watchObservedRunningTime="2026-11-02 08:05:02.405427119 +0000 UTC m=+2453.095754083"
Nov 02 08:05:03.373690 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:03.373655    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8" event={"ID":"5f970daf-e1a2-4859-8446-328a1d214d2e","Type":"ContainerStarted","Data":"eb51d89e7f50ea3f684782227381943f1742f7fddfb56f45dc3a9e1c415aae84"}
Nov 02 08:05:03.418505 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:03.418448    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8" podStartSLOduration=2.418433949 podStartE2EDuration="2.418433949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:03.415879052 +0000 UTC m=+2454.106206016" watchObservedRunningTime="2026-11-02 08:05:03.418433949 +0000 UTC m=+2454.108760929"
Nov 02 08:05:04.388191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:04.388151    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"
Nov 02 08:05:10.344674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:10.344644    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:05:10.424805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:10.423527    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"]
Nov 02 08:05:12.078434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:12.078395    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"]
Nov 02 08:05:12.572527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:12.572492    4898 scope.go:122] "RemoveContainer" containerID="64ca3b044bdcb97a94cbbf1997364304b4438b743c88439f11fc21d2bbf75135"
Nov 02 08:05:12.587099 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:12.587072    4898 scope.go:122] "RemoveContainer" containerID="fc8b7d473013a5a3a7d55203c662c8c27f928226253c0dc172cffe73595c0799"
Nov 02 08:05:12.597328 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:12.597302    4898 scope.go:122] "RemoveContainer" containerID="bb904bc56d309f027c9eef3edc45694447b6ab4e031771409a0b6427e0621f1c"
Nov 02 08:05:12.608776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:12.608747    4898 scope.go:122] "RemoveContainer" containerID="3f75ba4597a65fef8b75e43e6f0d34315628815a73e05cc091693a4d678f5cf0"
Nov 02 08:05:13.377620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:13.377588    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"
Nov 02 08:05:14.993156 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:05:14.993108    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:05:15.112086 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.112046    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"]
Nov 02 08:05:15.399904 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.399802    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"
Nov 02 08:05:15.424707 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.424661    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x" podUID="a7eb816c-6af2-472e-b255-8f5cc7f1099d" containerName="payload-processing" containerID="cri-o://7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53" gracePeriod=25
Nov 02 08:05:15.749374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.749344    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:05:15.865615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.865567    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-kube-api-access-kbqrl\" (UniqueName: \"kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-kube-api-access-kbqrl\") pod \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") "
Nov 02 08:05:15.865817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.865709    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-routing-overlay-volume\") pod \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") "
Nov 02 08:05:15.865817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.865778    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-plugins-config-volume\") pod \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") "
Nov 02 08:05:15.865918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.865848    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-provider-credentials\") pod \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\" (UID: \"a7eb816c-6af2-472e-b255-8f5cc7f1099d\") "
Nov 02 08:05:15.866222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.866192    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-routing-overlay-volume" pod "a7eb816c-6af2-472e-b255-8f5cc7f1099d" (UID: "a7eb816c-6af2-472e-b255-8f5cc7f1099d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:15.866630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.866600    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-plugins-config-volume" pod "a7eb816c-6af2-472e-b255-8f5cc7f1099d" (UID: "a7eb816c-6af2-472e-b255-8f5cc7f1099d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:15.866830 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.866809    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:15.866933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.866834    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb816c-6af2-472e-b255-8f5cc7f1099d-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:15.868627 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.868584    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-provider-credentials" pod "a7eb816c-6af2-472e-b255-8f5cc7f1099d" (UID: "a7eb816c-6af2-472e-b255-8f5cc7f1099d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:15.869324 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.869298    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-kube-api-access-kbqrl" pod "a7eb816c-6af2-472e-b255-8f5cc7f1099d" (UID: "a7eb816c-6af2-472e-b255-8f5cc7f1099d"). InnerVolumeSpecName "kube-api-access-kbqrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:15.968716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.968625    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:15.968716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:15.968669    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbqrl\" (UniqueName: \"kubernetes.io/projected/a7eb816c-6af2-472e-b255-8f5cc7f1099d-kube-api-access-kbqrl\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:16.522357 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:16.522323    4898 generic.go:356] "Generic (PLEG): container finished" podID="a7eb816c-6af2-472e-b255-8f5cc7f1099d" containerID="7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53" exitCode=0
Nov 02 08:05:16.522835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:16.522370    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"
Nov 02 08:05:16.522835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:16.522408    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x" event={"ID":"a7eb816c-6af2-472e-b255-8f5cc7f1099d","Type":"ContainerDied","Data":"7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53"}
Nov 02 08:05:16.522835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:16.522467    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x" event={"ID":"a7eb816c-6af2-472e-b255-8f5cc7f1099d","Type":"ContainerDied","Data":"7dd8bb9f113816d4f904263cf54b2c5e8aa51b3065792ab3847b7a07d92ee46d"}
Nov 02 08:05:16.522835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:16.522499    4898 scope.go:122] "RemoveContainer" containerID="7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53"
Nov 02 08:05:16.535958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:16.535935    4898 scope.go:122] "RemoveContainer" containerID="7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53"
Nov 02 08:05:16.536296 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:05:16.536259    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53\": container with ID starting with 7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53 not found: ID does not exist" containerID="7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53"
Nov 02 08:05:16.536372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:16.536309    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53"} err="failed to get container status \"7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53\": rpc error: code = NotFound desc = could not find container \"7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53\": container with ID starting with 7dd7a97124419382a2e33a5df9cfa7993dd6ea0ccdb1b3f4ac843804ec102a53 not found: ID does not exist"
Nov 02 08:05:16.576138 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:16.576100    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"]
Nov 02 08:05:16.588071 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:16.588038    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-76c8c769x"]
Nov 02 08:05:17.079865 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.079806    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg" podUID="c86d659e-732e-42e3-bbcd-f92548e031b9" containerName="payload-processing" containerID="cri-o://942f94cb2a2a573637976c3695e4d66fc3d79373d6390ba2b57a8a3ffef88251" gracePeriod=25
Nov 02 08:05:17.485826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.485788    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7eb816c-6af2-472e-b255-8f5cc7f1099d" path="/var/lib/kubelet/pods/a7eb816c-6af2-472e-b255-8f5cc7f1099d/volumes"
Nov 02 08:05:17.549019 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.548978    4898 generic.go:356] "Generic (PLEG): container finished" podID="c86d659e-732e-42e3-bbcd-f92548e031b9" containerID="942f94cb2a2a573637976c3695e4d66fc3d79373d6390ba2b57a8a3ffef88251" exitCode=0
Nov 02 08:05:17.549535 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.549258    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg" event={"ID":"c86d659e-732e-42e3-bbcd-f92548e031b9","Type":"ContainerDied","Data":"942f94cb2a2a573637976c3695e4d66fc3d79373d6390ba2b57a8a3ffef88251"}
Nov 02 08:05:17.549535 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.549313    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg" event={"ID":"c86d659e-732e-42e3-bbcd-f92548e031b9","Type":"ContainerDied","Data":"a57b4284d5adb791baaa70569435675e4ea04bb8425c2805de0cf5e43976d6d2"}
Nov 02 08:05:17.549535 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.549337    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a57b4284d5adb791baaa70569435675e4ea04bb8425c2805de0cf5e43976d6d2"
Nov 02 08:05:17.559707 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.559677    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:05:17.691913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.675076    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-provider-credentials\") pod \"c86d659e-732e-42e3-bbcd-f92548e031b9\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") "
Nov 02 08:05:17.691913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.675169    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-routing-overlay-volume\") pod \"c86d659e-732e-42e3-bbcd-f92548e031b9\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") "
Nov 02 08:05:17.691913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.675254    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-kube-api-access-z66lm\" (UniqueName: \"kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-kube-api-access-z66lm\") pod \"c86d659e-732e-42e3-bbcd-f92548e031b9\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") "
Nov 02 08:05:17.691913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.675489    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-plugins-config-volume\") pod \"c86d659e-732e-42e3-bbcd-f92548e031b9\" (UID: \"c86d659e-732e-42e3-bbcd-f92548e031b9\") "
Nov 02 08:05:17.722215 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.702134    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-routing-overlay-volume" pod "c86d659e-732e-42e3-bbcd-f92548e031b9" (UID: "c86d659e-732e-42e3-bbcd-f92548e031b9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:17.739938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.729030    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-plugins-config-volume" pod "c86d659e-732e-42e3-bbcd-f92548e031b9" (UID: "c86d659e-732e-42e3-bbcd-f92548e031b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:17.773546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.769609    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-provider-credentials" pod "c86d659e-732e-42e3-bbcd-f92548e031b9" (UID: "c86d659e-732e-42e3-bbcd-f92548e031b9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:17.794324 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.789368    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:17.794324 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.789406    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:17.794324 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.789420    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c86d659e-732e-42e3-bbcd-f92548e031b9-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:17.794324 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.789595    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-kube-api-access-z66lm" pod "c86d659e-732e-42e3-bbcd-f92548e031b9" (UID: "c86d659e-732e-42e3-bbcd-f92548e031b9"). InnerVolumeSpecName "kube-api-access-z66lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:17.891058 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:17.891015    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z66lm\" (UniqueName: \"kubernetes.io/projected/c86d659e-732e-42e3-bbcd-f92548e031b9-kube-api-access-z66lm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:18.210346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.210309    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"]
Nov 02 08:05:18.211129 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.211111    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a7eb816c-6af2-472e-b255-8f5cc7f1099d" containerName="payload-processing"
Nov 02 08:05:18.211188 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.211131    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7eb816c-6af2-472e-b255-8f5cc7f1099d" containerName="payload-processing"
Nov 02 08:05:18.211188 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.211148    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c86d659e-732e-42e3-bbcd-f92548e031b9" containerName="payload-processing"
Nov 02 08:05:18.211188 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.211154    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86d659e-732e-42e3-bbcd-f92548e031b9" containerName="payload-processing"
Nov 02 08:05:18.211288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.211266    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7eb816c-6af2-472e-b255-8f5cc7f1099d" containerName="payload-processing"
Nov 02 08:05:18.211288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.211277    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86d659e-732e-42e3-bbcd-f92548e031b9" containerName="payload-processing"
Nov 02 08:05:18.215113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.215089    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.240981 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.240950    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"]
Nov 02 08:05:18.296048 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.296011    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94898tkf\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.296229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.296131    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94898tkf\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.296339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.296318    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94898tkf\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.296379 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.296364    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtvbp\" (UniqueName: \"kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-kube-api-access-dtvbp\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94898tkf\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.397532 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.397485    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94898tkf\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.397727 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.397590    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94898tkf\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.397783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.397731    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94898tkf\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.397783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.397769    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtvbp\" (UniqueName: \"kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-kube-api-access-dtvbp\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94898tkf\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.398300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.398275    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94898tkf\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.398418 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.398311    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94898tkf\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.400222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.400199    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94898tkf\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.422716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.422676    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtvbp\" (UniqueName: \"kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-kube-api-access-dtvbp\") pod \"payload-processing-external-model-e2e-extmod-b-643355-b94898tkf\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.532861 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.532764    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:18.559727 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.559696    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"
Nov 02 08:05:18.636316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.634945    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"]
Nov 02 08:05:18.638931 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.638883    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587rhthg"]
Nov 02 08:05:18.850707 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:05:18.850674    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8797d27d_c6f3_4875_a688_4fbb9863ca65.slice/crio-45b323cdf33c702ec8f0d5e67362c4b0e627a4a278e995abcc1cd4af651ac2d9 WatchSource:0}: Error finding container 45b323cdf33c702ec8f0d5e67362c4b0e627a4a278e995abcc1cd4af651ac2d9: Status 404 returned error can't find the container with id 45b323cdf33c702ec8f0d5e67362c4b0e627a4a278e995abcc1cd4af651ac2d9
Nov 02 08:05:18.851678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:18.851650    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"]
Nov 02 08:05:19.468429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:19.468393    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c86d659e-732e-42e3-bbcd-f92548e031b9" path="/var/lib/kubelet/pods/c86d659e-732e-42e3-bbcd-f92548e031b9/volumes"
Nov 02 08:05:19.573008 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:19.572968    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf" event={"ID":"8797d27d-c6f3-4875-a688-4fbb9863ca65","Type":"ContainerStarted","Data":"5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d"}
Nov 02 08:05:19.573549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:19.573026    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf" event={"ID":"8797d27d-c6f3-4875-a688-4fbb9863ca65","Type":"ContainerStarted","Data":"45b323cdf33c702ec8f0d5e67362c4b0e627a4a278e995abcc1cd4af651ac2d9"}
Nov 02 08:05:19.573549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:19.573226    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:19.617379 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:19.617322    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf" podStartSLOduration=1.617308838 podStartE2EDuration="1.617308838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:19.613703674 +0000 UTC m=+2470.304030659" watchObservedRunningTime="2026-11-02 08:05:19.617308838 +0000 UTC m=+2470.307635801"
Nov 02 08:05:20.113430 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.113363    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq" podUID="71c5cea7-0dcd-4eea-ab45-d80f656f0861" containerName="payload-processing" containerID="cri-o://26229c12666dc74b4350165d062253eaacb34d4724016dbdffc6ffeb8b15a203" gracePeriod=25
Nov 02 08:05:20.699713 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.699602    4898 generic.go:356] "Generic (PLEG): container finished" podID="71c5cea7-0dcd-4eea-ab45-d80f656f0861" containerID="26229c12666dc74b4350165d062253eaacb34d4724016dbdffc6ffeb8b15a203" exitCode=0
Nov 02 08:05:20.700217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.699732    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq" event={"ID":"71c5cea7-0dcd-4eea-ab45-d80f656f0861","Type":"ContainerDied","Data":"26229c12666dc74b4350165d062253eaacb34d4724016dbdffc6ffeb8b15a203"}
Nov 02 08:05:20.742765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.742737    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:05:20.777709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.777669    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-provider-credentials\" (UniqueName: \"kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-provider-credentials\") pod \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") "
Nov 02 08:05:20.777937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.777756    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-kube-api-access-wzzrt\" (UniqueName: \"kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-kube-api-access-wzzrt\") pod \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") "
Nov 02 08:05:20.777937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.777853    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-routing-overlay-volume\") pod \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") "
Nov 02 08:05:20.778079 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.777956    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-plugins-config-volume\") pod \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\" (UID: \"71c5cea7-0dcd-4eea-ab45-d80f656f0861\") "
Nov 02 08:05:20.778310 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.778265    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-routing-overlay-volume" pod "71c5cea7-0dcd-4eea-ab45-d80f656f0861" (UID: "71c5cea7-0dcd-4eea-ab45-d80f656f0861"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:20.778435 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.778405    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-plugins-config-volume" pod "71c5cea7-0dcd-4eea-ab45-d80f656f0861" (UID: "71c5cea7-0dcd-4eea-ab45-d80f656f0861"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:20.779119 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.779093    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:20.779219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.779137    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c5cea7-0dcd-4eea-ab45-d80f656f0861-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:20.780626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.780604    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-kube-api-access-wzzrt" pod "71c5cea7-0dcd-4eea-ab45-d80f656f0861" (UID: "71c5cea7-0dcd-4eea-ab45-d80f656f0861"). InnerVolumeSpecName "kube-api-access-wzzrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:20.780700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.780637    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-provider-credentials" pod "71c5cea7-0dcd-4eea-ab45-d80f656f0861" (UID: "71c5cea7-0dcd-4eea-ab45-d80f656f0861"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:20.880087 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.880047    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:20.880087 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:20.880083    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzzrt\" (UniqueName: \"kubernetes.io/projected/71c5cea7-0dcd-4eea-ab45-d80f656f0861-kube-api-access-wzzrt\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:21.723816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:21.723724    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"
Nov 02 08:05:21.724382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:21.723731    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq" event={"ID":"71c5cea7-0dcd-4eea-ab45-d80f656f0861","Type":"ContainerDied","Data":"cd6bcd9fa5b236e2cb22df0c9df213c122b0757432e8b86af2cf674a22db4037"}
Nov 02 08:05:21.724382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:21.723925    4898 scope.go:122] "RemoveContainer" containerID="26229c12666dc74b4350165d062253eaacb34d4724016dbdffc6ffeb8b15a203"
Nov 02 08:05:21.783092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:21.783005    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"]
Nov 02 08:05:21.792769 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:21.792735    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-86967dxxq"]
Nov 02 08:05:23.273398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.273362    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"]
Nov 02 08:05:23.274109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.274092    4898 cpu_manager.go:414] "containerMap: removing container" podUID="71c5cea7-0dcd-4eea-ab45-d80f656f0861" containerName="payload-processing"
Nov 02 08:05:23.274109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.274108    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c5cea7-0dcd-4eea-ab45-d80f656f0861" containerName="payload-processing"
Nov 02 08:05:23.274268 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.274257    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c5cea7-0dcd-4eea-ab45-d80f656f0861" containerName="payload-processing"
Nov 02 08:05:23.279214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.279191    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.299153 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.299124    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"]
Nov 02 08:05:23.355522 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.355481    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.355730 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.355552    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.355730 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.355621    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.355730 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.355658    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g659w\" (UniqueName: \"kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-kube-api-access-g659w\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.457051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.457011    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.457276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.457093    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.457276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.457119    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.457276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.457142    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g659w\" (UniqueName: \"kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-kube-api-access-g659w\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.457945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.457869    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.457945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.457925    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.459782 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.459762    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.469321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.469283    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c5cea7-0dcd-4eea-ab45-d80f656f0861" path="/var/lib/kubelet/pods/71c5cea7-0dcd-4eea-ab45-d80f656f0861/volumes"
Nov 02 08:05:23.480974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.480948    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g659w\" (UniqueName: \"kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-kube-api-access-g659w\") pod \"payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") " pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.595454 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.595347    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:23.912047 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:05:23.912012    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ab319e_8c32_4ed7_8953_cb248bea36e6.slice/crio-9729e68c0bcaad25744181eddecb07f1711266a188c152f1195a20fb43f5d219 WatchSource:0}: Error finding container 9729e68c0bcaad25744181eddecb07f1711266a188c152f1195a20fb43f5d219: Status 404 returned error can't find the container with id 9729e68c0bcaad25744181eddecb07f1711266a188c152f1195a20fb43f5d219
Nov 02 08:05:23.912418 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:23.912392    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"]
Nov 02 08:05:24.473380 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.473345    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"]
Nov 02 08:05:24.478048 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.478025    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.506819 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.506785    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"agc-cred-56f589a0e742\""
Nov 02 08:05:24.507478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.507040    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"openshift-service-ca.crt\""
Nov 02 08:05:24.507478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.507143    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"routing-overlay\""
Nov 02 08:05:24.507478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.507170    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ea9556\""
Nov 02 08:05:24.507478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.507221    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"kube-root-ca.crt\""
Nov 02 08:05:24.507478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.507385    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"payload-processing-external-model-e2e--c1e7f36e-dockercfg-rl8k2\""
Nov 02 08:05:24.513654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.513620    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"]
Nov 02 08:05:24.613914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.595444    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.613914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.595617    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.613914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.595652    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4tls\" (UniqueName: \"kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-kube-api-access-w4tls\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.613914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.595685    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.698916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.696952    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.698916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.697108    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.698916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.697141    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4tls\" (UniqueName: \"kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-kube-api-access-w4tls\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.698916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.697176    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.698916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.698178    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.717794 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.717758    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.722694 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.722661    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.745493 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.743098    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4tls\" (UniqueName: \"kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-kube-api-access-w4tls\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.768212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.768172    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh" event={"ID":"e9ab319e-8c32-4ed7-8953-cb248bea36e6","Type":"ContainerStarted","Data":"1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0"}
Nov 02 08:05:24.768431 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.768242    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh" event={"ID":"e9ab319e-8c32-4ed7-8953-cb248bea36e6","Type":"ContainerStarted","Data":"9729e68c0bcaad25744181eddecb07f1711266a188c152f1195a20fb43f5d219"}
Nov 02 08:05:24.769021 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.769000    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:24.799469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.799427    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:24.810867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:24.810816    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh" podStartSLOduration=1.810802234 podStartE2EDuration="1.810802234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:24.809674521 +0000 UTC m=+2475.500001498" watchObservedRunningTime="2026-11-02 08:05:24.810802234 +0000 UTC m=+2475.501129198"
Nov 02 08:05:25.116455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.116425    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"]
Nov 02 08:05:25.116455 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:05:25.116445    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b36222_82e5_476d_9bd1_3d00f681d2cf.slice/crio-bfab111f9d91deba22a363ad9e999a66ad3f02f6c7a02d6edf4e7b60bc556de4 WatchSource:0}: Error finding container bfab111f9d91deba22a363ad9e999a66ad3f02f6c7a02d6edf4e7b60bc556de4: Status 404 returned error can't find the container with id bfab111f9d91deba22a363ad9e999a66ad3f02f6c7a02d6edf4e7b60bc556de4
Nov 02 08:05:25.141613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.141586    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"]
Nov 02 08:05:25.147377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.147357    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.163857 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.163814    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"agc-cred-2e46da42705f\""
Nov 02 08:05:25.164031 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.163964    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"payload-processing-external-model-e2e--99e06747-dockercfg-ssmlv\""
Nov 02 08:05:25.165900 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.165860    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"]
Nov 02 08:05:25.173051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.172453    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-71a41c\""
Nov 02 08:05:25.173051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.172468    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"kube-root-ca.crt\""
Nov 02 08:05:25.173051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.172535    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"routing-overlay\""
Nov 02 08:05:25.173051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.172590    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"openshift-service-ca.crt\""
Nov 02 08:05:25.173051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.172607    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"agc-cred-f93b3a589eb8\""
Nov 02 08:05:25.215862 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.215827    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.216222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.216191    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.216303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.216253    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xvdx\" (UniqueName: \"kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-kube-api-access-4xvdx\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.216375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.216354    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.317741 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.317697    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.317985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.317750    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xvdx\" (UniqueName: \"kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-kube-api-access-4xvdx\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.317985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.317790    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.317985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.317830    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.318584 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.318553    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.318686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.318607    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.321050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.320997    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.352463 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.352427    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xvdx\" (UniqueName: \"kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-kube-api-access-4xvdx\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.491251 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.491217    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:25.781870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.781787    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76" event={"ID":"d8b36222-82e5-476d-9bd1-3d00f681d2cf","Type":"ContainerStarted","Data":"5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f"}
Nov 02 08:05:25.781870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.781853    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76" event={"ID":"d8b36222-82e5-476d-9bd1-3d00f681d2cf","Type":"ContainerStarted","Data":"bfab111f9d91deba22a363ad9e999a66ad3f02f6c7a02d6edf4e7b60bc556de4"}
Nov 02 08:05:25.782122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.781898    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:25.822242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.822207    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"]
Nov 02 08:05:25.822515 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:05:25.822488    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62066a90_e562_4670_b48f_0b0ae1c5de7e.slice/crio-fc04c3f69e2a2070bc0910d997375771f9c6b230faa495cb62332d0b8b8eaec1 WatchSource:0}: Error finding container fc04c3f69e2a2070bc0910d997375771f9c6b230faa495cb62332d0b8b8eaec1: Status 404 returned error can't find the container with id fc04c3f69e2a2070bc0910d997375771f9c6b230faa495cb62332d0b8b8eaec1
Nov 02 08:05:25.833560 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:25.833504    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76" podStartSLOduration=1.83348646 podStartE2EDuration="1.83348646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:25.831461434 +0000 UTC m=+2476.521788402" watchObservedRunningTime="2026-11-02 08:05:25.83348646 +0000 UTC m=+2476.523813425"
Nov 02 08:05:26.792759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:26.792720    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m" event={"ID":"62066a90-e562-4670-b48f-0b0ae1c5de7e","Type":"ContainerStarted","Data":"34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601"}
Nov 02 08:05:26.793289 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:26.792786    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m" event={"ID":"62066a90-e562-4670-b48f-0b0ae1c5de7e","Type":"ContainerStarted","Data":"fc04c3f69e2a2070bc0910d997375771f9c6b230faa495cb62332d0b8b8eaec1"}
Nov 02 08:05:26.838956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:26.838897    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m" podStartSLOduration=1.8388717319999999 podStartE2EDuration="1.838871732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:26.833999746 +0000 UTC m=+2477.524326735" watchObservedRunningTime="2026-11-02 08:05:26.838871732 +0000 UTC m=+2477.529198696"
Nov 02 08:05:26.935537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:26.935498    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"]
Nov 02 08:05:26.938812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:26.938782    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:26.967561 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:26.967525    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"]
Nov 02 08:05:26.972482 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:26.972457    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:26.995028 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:26.994990    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"]
Nov 02 08:05:27.002899 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:26.998476    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-643355\"/\"agc-cred-ee45b14f84b8\""
Nov 02 08:05:27.048671 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.048567    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.048856 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.048675    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.048856 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.048787    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwvq\" (UniqueName: \"kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-kube-api-access-dzwvq\") pod \"payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.049051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.049028    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.149938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.149849    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.150120 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.150050    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.150120 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.150104    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.150215 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.150203    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwvq\" (UniqueName: \"kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-kube-api-access-dzwvq\") pod \"payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.150872 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.150843    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.151025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.150949    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.152544 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.152521    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.174320 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.174283    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwvq\" (UniqueName: \"kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-kube-api-access-dzwvq\") pod \"payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") " pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.294212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.294163    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.644752 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:05:27.644718    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode301fa34_3c7e_4b26_bef8_612a0925fb2b.slice/crio-7c38906e89dc977f93c364edd82646881b599e31a9a4400805f4547ab923123c WatchSource:0}: Error finding container 7c38906e89dc977f93c364edd82646881b599e31a9a4400805f4547ab923123c: Status 404 returned error can't find the container with id 7c38906e89dc977f93c364edd82646881b599e31a9a4400805f4547ab923123c
Nov 02 08:05:27.651978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.651951    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"]
Nov 02 08:05:27.805591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.805547    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4" event={"ID":"e301fa34-3c7e-4b26-bef8-612a0925fb2b","Type":"ContainerStarted","Data":"efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8"}
Nov 02 08:05:27.806161 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.805647    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4" event={"ID":"e301fa34-3c7e-4b26-bef8-612a0925fb2b","Type":"ContainerStarted","Data":"7c38906e89dc977f93c364edd82646881b599e31a9a4400805f4547ab923123c"}
Nov 02 08:05:27.806161 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.805883    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:27.806161 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.805968    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:27.850009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:27.849952    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4" podStartSLOduration=1.849938414 podStartE2EDuration="1.849938414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:27.848668783 +0000 UTC m=+2478.538995748" watchObservedRunningTime="2026-11-02 08:05:27.849938414 +0000 UTC m=+2478.540265405"
Nov 02 08:05:28.341985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:28.341948    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"]
Nov 02 08:05:30.626518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.626482    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"]
Nov 02 08:05:30.632573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.632547    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:30.651273 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.650448    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:05:30.651273 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.650490    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:05:30.651273 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.650546    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:05:30.652136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.652108    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"]
Nov 02 08:05:30.669621 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.669585    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:05:30.727716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.727677    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77714a05-8jncb\" (UID: \"e95d37ea-b122-4c23-beed-9e888217d3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:30.727957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.727871    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77714a05-8jncb\" (UID: \"e95d37ea-b122-4c23-beed-9e888217d3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:30.727957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.727943    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kkcv\" (UniqueName: \"kubernetes.io/projected/e95d37ea-b122-4c23-beed-9e888217d3bf-kube-api-access-8kkcv\") pod \"maas-api-revoke-sub-77714a05-8jncb\" (UID: \"e95d37ea-b122-4c23-beed-9e888217d3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:30.829457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.829409    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77714a05-8jncb\" (UID: \"e95d37ea-b122-4c23-beed-9e888217d3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:30.829457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.829463    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kkcv\" (UniqueName: \"kubernetes.io/projected/e95d37ea-b122-4c23-beed-9e888217d3bf-kube-api-access-8kkcv\") pod \"maas-api-revoke-sub-77714a05-8jncb\" (UID: \"e95d37ea-b122-4c23-beed-9e888217d3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:30.829740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.829612    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77714a05-8jncb\" (UID: \"e95d37ea-b122-4c23-beed-9e888217d3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:30.830161 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.830127    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77714a05-8jncb\" (UID: \"e95d37ea-b122-4c23-beed-9e888217d3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:30.832180 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.832151    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77714a05-8jncb\" (UID: \"e95d37ea-b122-4c23-beed-9e888217d3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:30.853798 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.853752    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kkcv\" (UniqueName: \"kubernetes.io/projected/e95d37ea-b122-4c23-beed-9e888217d3bf-kube-api-access-8kkcv\") pod \"maas-api-revoke-sub-77714a05-8jncb\" (UID: \"e95d37ea-b122-4c23-beed-9e888217d3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:30.990924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:30.990858    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:31.302581 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:05:31.302538    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95d37ea_b122_4c23_beed_9e888217d3bf.slice/crio-63d77909d4e3c3a1d3cd84f631f45f4d226604f7731f4576c070eace3bd0420c WatchSource:0}: Error finding container 63d77909d4e3c3a1d3cd84f631f45f4d226604f7731f4576c070eace3bd0420c: Status 404 returned error can't find the container with id 63d77909d4e3c3a1d3cd84f631f45f4d226604f7731f4576c070eace3bd0420c
Nov 02 08:05:31.302732 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:31.302676    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"]
Nov 02 08:05:31.901050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:31.901004    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb" event={"ID":"e95d37ea-b122-4c23-beed-9e888217d3bf","Type":"ContainerStarted","Data":"fa2e5d4d89a0f80ea62271bbeab0b28b5a1f299c44641068d8ab8a8d190b9e72"}
Nov 02 08:05:31.901562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:31.901078    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb" event={"ID":"e95d37ea-b122-4c23-beed-9e888217d3bf","Type":"ContainerStarted","Data":"63d77909d4e3c3a1d3cd84f631f45f4d226604f7731f4576c070eace3bd0420c"}
Nov 02 08:05:31.936300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:31.936200    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf" podUID="8797d27d-c6f3-4875-a688-4fbb9863ca65" containerName="payload-processing" containerID="cri-o://5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d" gracePeriod=25
Nov 02 08:05:31.942597 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:31.942548    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb" podStartSLOduration=1.942534714 podStartE2EDuration="1.942534714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:31.942305354 +0000 UTC m=+2482.632632338" watchObservedRunningTime="2026-11-02 08:05:31.942534714 +0000 UTC m=+2482.632861677"
Nov 02 08:05:32.235465 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.235434    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:32.354317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.354282    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-plugins-config-volume\") pod \"8797d27d-c6f3-4875-a688-4fbb9863ca65\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") "
Nov 02 08:05:32.354514 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.354497    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-provider-credentials\") pod \"8797d27d-c6f3-4875-a688-4fbb9863ca65\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") "
Nov 02 08:05:32.354585 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.354551    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-kube-api-access-dtvbp\" (UniqueName: \"kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-kube-api-access-dtvbp\") pod \"8797d27d-c6f3-4875-a688-4fbb9863ca65\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") "
Nov 02 08:05:32.354641 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.354592    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-routing-overlay-volume\") pod \"8797d27d-c6f3-4875-a688-4fbb9863ca65\" (UID: \"8797d27d-c6f3-4875-a688-4fbb9863ca65\") "
Nov 02 08:05:32.354700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.354667    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-plugins-config-volume" pod "8797d27d-c6f3-4875-a688-4fbb9863ca65" (UID: "8797d27d-c6f3-4875-a688-4fbb9863ca65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:32.355224 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.355197    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:32.355435 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.355414    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-routing-overlay-volume" pod "8797d27d-c6f3-4875-a688-4fbb9863ca65" (UID: "8797d27d-c6f3-4875-a688-4fbb9863ca65"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:32.356794 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.356769    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-provider-credentials" pod "8797d27d-c6f3-4875-a688-4fbb9863ca65" (UID: "8797d27d-c6f3-4875-a688-4fbb9863ca65"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:32.356936 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.356789    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-kube-api-access-dtvbp" pod "8797d27d-c6f3-4875-a688-4fbb9863ca65" (UID: "8797d27d-c6f3-4875-a688-4fbb9863ca65"). InnerVolumeSpecName "kube-api-access-dtvbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:32.456923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.456815    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:32.456923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.456846    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtvbp\" (UniqueName: \"kubernetes.io/projected/8797d27d-c6f3-4875-a688-4fbb9863ca65-kube-api-access-dtvbp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:32.456923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.456857    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8797d27d-c6f3-4875-a688-4fbb9863ca65-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:32.911639 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.911600    4898 generic.go:356] "Generic (PLEG): container finished" podID="8797d27d-c6f3-4875-a688-4fbb9863ca65" containerID="5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d" exitCode=0
Nov 02 08:05:32.912146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.911644    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"
Nov 02 08:05:32.912146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.911688    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf" event={"ID":"8797d27d-c6f3-4875-a688-4fbb9863ca65","Type":"ContainerDied","Data":"5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d"}
Nov 02 08:05:32.912146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.911760    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf" event={"ID":"8797d27d-c6f3-4875-a688-4fbb9863ca65","Type":"ContainerDied","Data":"45b323cdf33c702ec8f0d5e67362c4b0e627a4a278e995abcc1cd4af651ac2d9"}
Nov 02 08:05:32.912146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.911794    4898 scope.go:122] "RemoveContainer" containerID="5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d"
Nov 02 08:05:32.914036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.914007    4898 generic.go:356] "Generic (PLEG): container finished" podID="e95d37ea-b122-4c23-beed-9e888217d3bf" containerID="fa2e5d4d89a0f80ea62271bbeab0b28b5a1f299c44641068d8ab8a8d190b9e72" exitCode=0
Nov 02 08:05:32.914147 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.914093    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb" event={"ID":"e95d37ea-b122-4c23-beed-9e888217d3bf","Type":"ContainerDied","Data":"fa2e5d4d89a0f80ea62271bbeab0b28b5a1f299c44641068d8ab8a8d190b9e72"}
Nov 02 08:05:32.925098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.925081    4898 scope.go:122] "RemoveContainer" containerID="5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d"
Nov 02 08:05:32.925381 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:05:32.925359    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d\": container with ID starting with 5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d not found: ID does not exist" containerID="5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d"
Nov 02 08:05:32.925422 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:32.925389    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d"} err="failed to get container status \"5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d\": rpc error: code = NotFound desc = could not find container \"5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d\": container with ID starting with 5ced974dce0b575ba5716971c80c8a26e961ebd2d4bf80b732e5e7d9d583824d not found: ID does not exist"
Nov 02 08:05:33.015976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.015945    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"]
Nov 02 08:05:33.022509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.022476    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-b94898tkf"]
Nov 02 08:05:33.342312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.342253    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76" podUID="d8b36222-82e5-476d-9bd1-3d00f681d2cf" containerName="payload-processing" containerID="cri-o://5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f" gracePeriod=25
Nov 02 08:05:33.479271 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.479238    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8797d27d-c6f3-4875-a688-4fbb9863ca65" path="/var/lib/kubelet/pods/8797d27d-c6f3-4875-a688-4fbb9863ca65/volumes"
Nov 02 08:05:33.668518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.668482    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:33.778500 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.778464    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-plugins-config-volume\") pod \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") "
Nov 02 08:05:33.778689 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.778560    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-provider-credentials\") pod \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") "
Nov 02 08:05:33.778751 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.778734    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-routing-overlay-volume\") pod \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") "
Nov 02 08:05:33.778845 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.778828    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-kube-api-access-w4tls\" (UniqueName: \"kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-kube-api-access-w4tls\") pod \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\" (UID: \"d8b36222-82e5-476d-9bd1-3d00f681d2cf\") "
Nov 02 08:05:33.779126 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.779092    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-plugins-config-volume" pod "d8b36222-82e5-476d-9bd1-3d00f681d2cf" (UID: "d8b36222-82e5-476d-9bd1-3d00f681d2cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:33.779276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.779155    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-routing-overlay-volume" pod "d8b36222-82e5-476d-9bd1-3d00f681d2cf" (UID: "d8b36222-82e5-476d-9bd1-3d00f681d2cf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:33.779639 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.779613    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:33.779734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.779644    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b36222-82e5-476d-9bd1-3d00f681d2cf-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:33.781207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.781180    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-provider-credentials" pod "d8b36222-82e5-476d-9bd1-3d00f681d2cf" (UID: "d8b36222-82e5-476d-9bd1-3d00f681d2cf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:33.781295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.781214    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-kube-api-access-w4tls" pod "d8b36222-82e5-476d-9bd1-3d00f681d2cf" (UID: "d8b36222-82e5-476d-9bd1-3d00f681d2cf"). InnerVolumeSpecName "kube-api-access-w4tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:33.881918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.881816    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:33.881918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.881861    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4tls\" (UniqueName: \"kubernetes.io/projected/d8b36222-82e5-476d-9bd1-3d00f681d2cf-kube-api-access-w4tls\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:33.925350 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.925315    4898 generic.go:356] "Generic (PLEG): container finished" podID="d8b36222-82e5-476d-9bd1-3d00f681d2cf" containerID="5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f" exitCode=0
Nov 02 08:05:33.925827 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.925376    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"
Nov 02 08:05:33.925827 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.925409    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76" event={"ID":"d8b36222-82e5-476d-9bd1-3d00f681d2cf","Type":"ContainerDied","Data":"5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f"}
Nov 02 08:05:33.925827 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.925468    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76" event={"ID":"d8b36222-82e5-476d-9bd1-3d00f681d2cf","Type":"ContainerDied","Data":"bfab111f9d91deba22a363ad9e999a66ad3f02f6c7a02d6edf4e7b60bc556de4"}
Nov 02 08:05:33.925827 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.925493    4898 scope.go:122] "RemoveContainer" containerID="5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f"
Nov 02 08:05:33.939210 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.939192    4898 scope.go:122] "RemoveContainer" containerID="5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f"
Nov 02 08:05:33.939559 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:05:33.939535    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f\": container with ID starting with 5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f not found: ID does not exist" containerID="5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f"
Nov 02 08:05:33.939714 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.939564    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f"} err="failed to get container status \"5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f\": rpc error: code = NotFound desc = could not find container \"5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f\": container with ID starting with 5612747f0fa4ca029a6e1391eef699abf15c5c67691ec8f0e4c671877d41200f not found: ID does not exist"
Nov 02 08:05:33.986608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.986578    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"]
Nov 02 08:05:33.994203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:33.992138    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-59f7z6f76"]
Nov 02 08:05:34.113552 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.113524    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:34.287861 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.287827    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95d37ea-b122-4c23-beed-9e888217d3bf-kube-api-access-8kkcv\" (UniqueName: \"kubernetes.io/projected/e95d37ea-b122-4c23-beed-9e888217d3bf-kube-api-access-8kkcv\") pod \"e95d37ea-b122-4c23-beed-9e888217d3bf\" (UID: \"e95d37ea-b122-4c23-beed-9e888217d3bf\") "
Nov 02 08:05:34.288089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.287934    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-xks-ca\") pod \"e95d37ea-b122-4c23-beed-9e888217d3bf\" (UID: \"e95d37ea-b122-4c23-beed-9e888217d3bf\") "
Nov 02 08:05:34.288089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.287963    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-openshift-ca\") pod \"e95d37ea-b122-4c23-beed-9e888217d3bf\" (UID: \"e95d37ea-b122-4c23-beed-9e888217d3bf\") "
Nov 02 08:05:34.288398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.288371    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-openshift-ca" pod "e95d37ea-b122-4c23-beed-9e888217d3bf" (UID: "e95d37ea-b122-4c23-beed-9e888217d3bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:34.288509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.288495    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:34.290131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.290100    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-xks-ca" pod "e95d37ea-b122-4c23-beed-9e888217d3bf" (UID: "e95d37ea-b122-4c23-beed-9e888217d3bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:05:34.290247 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.290165    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95d37ea-b122-4c23-beed-9e888217d3bf-kube-api-access-8kkcv" pod "e95d37ea-b122-4c23-beed-9e888217d3bf" (UID: "e95d37ea-b122-4c23-beed-9e888217d3bf"). InnerVolumeSpecName "kube-api-access-8kkcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:34.390464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.390428    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kkcv\" (UniqueName: \"kubernetes.io/projected/e95d37ea-b122-4c23-beed-9e888217d3bf-kube-api-access-8kkcv\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:34.390464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.390460    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95d37ea-b122-4c23-beed-9e888217d3bf-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:34.934187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.934158    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"
Nov 02 08:05:34.934653 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.934150    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb" event={"ID":"e95d37ea-b122-4c23-beed-9e888217d3bf","Type":"ContainerDied","Data":"63d77909d4e3c3a1d3cd84f631f45f4d226604f7731f4576c070eace3bd0420c"}
Nov 02 08:05:34.934653 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:34.934306    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63d77909d4e3c3a1d3cd84f631f45f4d226604f7731f4576c070eace3bd0420c"
Nov 02 08:05:35.467157 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:35.467117    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b36222-82e5-476d-9bd1-3d00f681d2cf" path="/var/lib/kubelet/pods/d8b36222-82e5-476d-9bd1-3d00f681d2cf/volumes"
Nov 02 08:05:35.786156 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:35.786066    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:05:35.862171 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:35.862101    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"]
Nov 02 08:05:38.847462 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:38.847432    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:05:38.847898 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:38.847810    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:05:40.862963 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:40.862910    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq" podUID="6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b" containerName="payload-processing" containerID="cri-o://4ec46ef1640270f743a7243ef5ccd3bf3edd373a73ff6f7b55df81f423cbcdc7" gracePeriod=25
Nov 02 08:05:41.006638 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.006606    4898 generic.go:356] "Generic (PLEG): container finished" podID="6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b" containerID="4ec46ef1640270f743a7243ef5ccd3bf3edd373a73ff6f7b55df81f423cbcdc7" exitCode=0
Nov 02 08:05:41.006816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.006679    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq" event={"ID":"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b","Type":"ContainerDied","Data":"4ec46ef1640270f743a7243ef5ccd3bf3edd373a73ff6f7b55df81f423cbcdc7"}
Nov 02 08:05:41.158285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.158254    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:05:41.202541 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.202505    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-routing-overlay-volume\") pod \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") "
Nov 02 08:05:41.202817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.202791    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-provider-credentials\") pod \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") "
Nov 02 08:05:41.203276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.202865    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-kube-api-access-4ss97\" (UniqueName: \"kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-kube-api-access-4ss97\") pod \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") "
Nov 02 08:05:41.203276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.203045    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-plugins-config-volume\") pod \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\" (UID: \"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b\") "
Nov 02 08:05:41.203276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.203192    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-routing-overlay-volume" pod "6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b" (UID: "6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:41.204009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.203664    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:41.204009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.203975    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-plugins-config-volume" pod "6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b" (UID: "6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:41.205638 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.205601    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-kube-api-access-4ss97" pod "6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b" (UID: "6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b"). InnerVolumeSpecName "kube-api-access-4ss97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:41.205853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.205826    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-provider-credentials" pod "6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b" (UID: "6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:41.271698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.271665    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"]
Nov 02 08:05:41.272518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.272495    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d8b36222-82e5-476d-9bd1-3d00f681d2cf" containerName="payload-processing"
Nov 02 08:05:41.272518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.272517    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b36222-82e5-476d-9bd1-3d00f681d2cf" containerName="payload-processing"
Nov 02 08:05:41.272734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.272540    4898 cpu_manager.go:414] "containerMap: removing container" podUID="8797d27d-c6f3-4875-a688-4fbb9863ca65" containerName="payload-processing"
Nov 02 08:05:41.272734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.272547    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8797d27d-c6f3-4875-a688-4fbb9863ca65" containerName="payload-processing"
Nov 02 08:05:41.272734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.272557    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e95d37ea-b122-4c23-beed-9e888217d3bf" containerName="revoke-keys"
Nov 02 08:05:41.272734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.272564    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95d37ea-b122-4c23-beed-9e888217d3bf" containerName="revoke-keys"
Nov 02 08:05:41.272734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.272642    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b" containerName="payload-processing"
Nov 02 08:05:41.272734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.272649    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b" containerName="payload-processing"
Nov 02 08:05:41.273113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.272844    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8797d27d-c6f3-4875-a688-4fbb9863ca65" containerName="payload-processing"
Nov 02 08:05:41.273113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.272857    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95d37ea-b122-4c23-beed-9e888217d3bf" containerName="revoke-keys"
Nov 02 08:05:41.273113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.272884    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b" containerName="payload-processing"
Nov 02 08:05:41.273113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.272914    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b36222-82e5-476d-9bd1-3d00f681d2cf" containerName="payload-processing"
Nov 02 08:05:41.276820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.276801    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:41.292114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.291874    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:05:41.292114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.292032    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:05:41.292114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.292033    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:05:41.292114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.292093    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:05:41.294782 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.294762    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"]
Nov 02 08:05:41.305838 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.305803    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e8be127-wtnmw\" (UID: \"78356009-a9e0-404d-a82b-0518bfa85ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:41.307269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.307238    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e8be127-wtnmw\" (UID: \"78356009-a9e0-404d-a82b-0518bfa85ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:41.307378 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.307346    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkt6l\" (UniqueName: \"kubernetes.io/projected/78356009-a9e0-404d-a82b-0518bfa85ba3-kube-api-access-mkt6l\") pod \"maas-api-revoke-sub-4e8be127-wtnmw\" (UID: \"78356009-a9e0-404d-a82b-0518bfa85ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:41.307490 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.307473    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:41.307527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.307498    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ss97\" (UniqueName: \"kubernetes.io/projected/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-kube-api-access-4ss97\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:41.307576 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.307563    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:41.409189 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.409155    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e8be127-wtnmw\" (UID: \"78356009-a9e0-404d-a82b-0518bfa85ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:41.409377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.409226    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkt6l\" (UniqueName: \"kubernetes.io/projected/78356009-a9e0-404d-a82b-0518bfa85ba3-kube-api-access-mkt6l\") pod \"maas-api-revoke-sub-4e8be127-wtnmw\" (UID: \"78356009-a9e0-404d-a82b-0518bfa85ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:41.409377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.409334    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e8be127-wtnmw\" (UID: \"78356009-a9e0-404d-a82b-0518bfa85ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:41.409975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.409947    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e8be127-wtnmw\" (UID: \"78356009-a9e0-404d-a82b-0518bfa85ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:41.411692 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.411671    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e8be127-wtnmw\" (UID: \"78356009-a9e0-404d-a82b-0518bfa85ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:41.432454 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.432430    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkt6l\" (UniqueName: \"kubernetes.io/projected/78356009-a9e0-404d-a82b-0518bfa85ba3-kube-api-access-mkt6l\") pod \"maas-api-revoke-sub-4e8be127-wtnmw\" (UID: \"78356009-a9e0-404d-a82b-0518bfa85ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:41.599818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.599784    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:41.908883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:41.908852    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"]
Nov 02 08:05:41.909435 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:05:41.909410    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78356009_a9e0_404d_a82b_0518bfa85ba3.slice/crio-a87c308a6093140d9993c6faccd366873ad0a5f2e34b183a51bf4ba19e837687 WatchSource:0}: Error finding container a87c308a6093140d9993c6faccd366873ad0a5f2e34b183a51bf4ba19e837687: Status 404 returned error can't find the container with id a87c308a6093140d9993c6faccd366873ad0a5f2e34b183a51bf4ba19e837687
Nov 02 08:05:42.015774 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:42.015745    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"
Nov 02 08:05:42.015963 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:42.015787    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq" event={"ID":"6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b","Type":"ContainerDied","Data":"79846ced2fb1223d27e72556fbafc82bd9e9700cadc26eef0c0d259525106c61"}
Nov 02 08:05:42.015963 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:42.015849    4898 scope.go:122] "RemoveContainer" containerID="4ec46ef1640270f743a7243ef5ccd3bf3edd373a73ff6f7b55df81f423cbcdc7"
Nov 02 08:05:42.017484 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:42.017455    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw" event={"ID":"78356009-a9e0-404d-a82b-0518bfa85ba3","Type":"ContainerStarted","Data":"a87c308a6093140d9993c6faccd366873ad0a5f2e34b183a51bf4ba19e837687"}
Nov 02 08:05:42.073361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:42.073330    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"]
Nov 02 08:05:42.077285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:42.077259    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-6896wldzq"]
Nov 02 08:05:43.034565 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:43.034528    4898 generic.go:356] "Generic (PLEG): container finished" podID="78356009-a9e0-404d-a82b-0518bfa85ba3" containerID="1d08934ddfac5213d129c6bf1e229221733bc633328e7edf6ce052c70577fc19" exitCode=0
Nov 02 08:05:43.035066 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:43.034614    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw" event={"ID":"78356009-a9e0-404d-a82b-0518bfa85ba3","Type":"ContainerDied","Data":"1d08934ddfac5213d129c6bf1e229221733bc633328e7edf6ce052c70577fc19"}
Nov 02 08:05:43.466451 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:43.466417    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b" path="/var/lib/kubelet/pods/6ae1a5c9-ebcc-470e-b20a-ce3d9df5eb2b/volumes"
Nov 02 08:05:44.218570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:44.218536    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:44.256432 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:44.256399    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-xks-ca\") pod \"78356009-a9e0-404d-a82b-0518bfa85ba3\" (UID: \"78356009-a9e0-404d-a82b-0518bfa85ba3\") "
Nov 02 08:05:44.256645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:44.256617    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-openshift-ca\") pod \"78356009-a9e0-404d-a82b-0518bfa85ba3\" (UID: \"78356009-a9e0-404d-a82b-0518bfa85ba3\") "
Nov 02 08:05:44.256822 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:44.256801    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78356009-a9e0-404d-a82b-0518bfa85ba3-kube-api-access-mkt6l\" (UniqueName: \"kubernetes.io/projected/78356009-a9e0-404d-a82b-0518bfa85ba3-kube-api-access-mkt6l\") pod \"78356009-a9e0-404d-a82b-0518bfa85ba3\" (UID: \"78356009-a9e0-404d-a82b-0518bfa85ba3\") "
Nov 02 08:05:44.257053 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:44.257026    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-openshift-ca" pod "78356009-a9e0-404d-a82b-0518bfa85ba3" (UID: "78356009-a9e0-404d-a82b-0518bfa85ba3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:44.258336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:44.257470    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:44.259835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:44.259794    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78356009-a9e0-404d-a82b-0518bfa85ba3-kube-api-access-mkt6l" pod "78356009-a9e0-404d-a82b-0518bfa85ba3" (UID: "78356009-a9e0-404d-a82b-0518bfa85ba3"). InnerVolumeSpecName "kube-api-access-mkt6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:44.261250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:44.261215    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-xks-ca" pod "78356009-a9e0-404d-a82b-0518bfa85ba3" (UID: "78356009-a9e0-404d-a82b-0518bfa85ba3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:05:44.364973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:44.359810    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkt6l\" (UniqueName: \"kubernetes.io/projected/78356009-a9e0-404d-a82b-0518bfa85ba3-kube-api-access-mkt6l\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:44.364973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:44.359854    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78356009-a9e0-404d-a82b-0518bfa85ba3-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:44.993245 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:05:44.993197    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:05:45.053467 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:45.053434    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"
Nov 02 08:05:45.053619 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:45.053464    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw" event={"ID":"78356009-a9e0-404d-a82b-0518bfa85ba3","Type":"ContainerDied","Data":"a87c308a6093140d9993c6faccd366873ad0a5f2e34b183a51bf4ba19e837687"}
Nov 02 08:05:45.053619 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:45.053515    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87c308a6093140d9993c6faccd366873ad0a5f2e34b183a51bf4ba19e837687"
Nov 02 08:05:54.199560 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.199520    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"]
Nov 02 08:05:54.200343 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.200324    4898 cpu_manager.go:414] "containerMap: removing container" podUID="78356009-a9e0-404d-a82b-0518bfa85ba3" containerName="revoke-keys"
Nov 02 08:05:54.200343 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.200344    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78356009-a9e0-404d-a82b-0518bfa85ba3" containerName="revoke-keys"
Nov 02 08:05:54.200460 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.200451    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="78356009-a9e0-404d-a82b-0518bfa85ba3" containerName="revoke-keys"
Nov 02 08:05:54.204103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.204084    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:54.239826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.239790    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:05:54.240251 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.240230    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:05:54.240458 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.240442    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:05:54.240622 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.240607    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:05:54.256111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.256056    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"]
Nov 02 08:05:54.364469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.364421    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-576a953c-j27kh\" (UID: \"f526e30c-847e-4602-b4b1-f272c4e46c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:54.364655 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.364637    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-576a953c-j27kh\" (UID: \"f526e30c-847e-4602-b4b1-f272c4e46c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:54.364701 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.364669    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pspl8\" (UniqueName: \"kubernetes.io/projected/f526e30c-847e-4602-b4b1-f272c4e46c96-kube-api-access-pspl8\") pod \"maas-api-revoke-sub-576a953c-j27kh\" (UID: \"f526e30c-847e-4602-b4b1-f272c4e46c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:54.466477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.466390    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-576a953c-j27kh\" (UID: \"f526e30c-847e-4602-b4b1-f272c4e46c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:54.466477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.466431    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pspl8\" (UniqueName: \"kubernetes.io/projected/f526e30c-847e-4602-b4b1-f272c4e46c96-kube-api-access-pspl8\") pod \"maas-api-revoke-sub-576a953c-j27kh\" (UID: \"f526e30c-847e-4602-b4b1-f272c4e46c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:54.466707 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.466576    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-576a953c-j27kh\" (UID: \"f526e30c-847e-4602-b4b1-f272c4e46c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:54.467096 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.467069    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-576a953c-j27kh\" (UID: \"f526e30c-847e-4602-b4b1-f272c4e46c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:54.469252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.469225    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-576a953c-j27kh\" (UID: \"f526e30c-847e-4602-b4b1-f272c4e46c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:54.490597 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.490568    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pspl8\" (UniqueName: \"kubernetes.io/projected/f526e30c-847e-4602-b4b1-f272c4e46c96-kube-api-access-pspl8\") pod \"maas-api-revoke-sub-576a953c-j27kh\" (UID: \"f526e30c-847e-4602-b4b1-f272c4e46c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:54.530557 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.530527    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:54.844427 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:05:54.844384    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf526e30c_847e_4602_b4b1_f272c4e46c96.slice/crio-aee379ce576ff5c84676ab6155c8329613993ae0317fe298b7c7169baa09924e WatchSource:0}: Error finding container aee379ce576ff5c84676ab6155c8329613993ae0317fe298b7c7169baa09924e: Status 404 returned error can't find the container with id aee379ce576ff5c84676ab6155c8329613993ae0317fe298b7c7169baa09924e
Nov 02 08:05:54.845258 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:54.845227    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"]
Nov 02 08:05:55.156323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.156289    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh" event={"ID":"f526e30c-847e-4602-b4b1-f272c4e46c96","Type":"ContainerStarted","Data":"aee379ce576ff5c84676ab6155c8329613993ae0317fe298b7c7169baa09924e"}
Nov 02 08:05:55.668033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.667999    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"]
Nov 02 08:05:55.672478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.672453    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:55.692417 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.692386    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"]
Nov 02 08:05:55.700193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.699577    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:55.700193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.699677    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"agc-cred-74b36e68d5bd\""
Nov 02 08:05:55.700193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.699716    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrp6\" (UniqueName: \"kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-kube-api-access-rxrp6\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:55.700193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.699733    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"agc-cred-c91e3cdca9f0\""
Nov 02 08:05:55.700193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.699768    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"agc-cred-0687728475e9\""
Nov 02 08:05:55.700193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.699782    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:55.700193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.699855    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2dde84\""
Nov 02 08:05:55.700193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.699589    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-e2e--1a3816b7-dockercfg-4nccn\""
Nov 02 08:05:55.700193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.700020    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:55.700923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.700863    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"openshift-service-ca.crt\""
Nov 02 08:05:55.701030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.701012    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"routing-overlay\""
Nov 02 08:05:55.701097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.701088    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"kube-root-ca.crt\""
Nov 02 08:05:55.801841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.801801    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:55.802051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.801951    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:55.802051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.802046    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrp6\" (UniqueName: \"kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-kube-api-access-rxrp6\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:55.802149 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.802093    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:55.802748 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.802716    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:55.802914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.802868    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:55.804511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.804489    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:55.825822 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:55.825793    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrp6\" (UniqueName: \"kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-kube-api-access-rxrp6\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:56.073961 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:56.073369    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:56.193761 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:56.193719    4898 generic.go:356] "Generic (PLEG): container finished" podID="f526e30c-847e-4602-b4b1-f272c4e46c96" containerID="f9911eb9656b24451cdb0967b31c0b817aced8e735c54a3d83e3d0fd2b93986e" exitCode=0
Nov 02 08:05:56.195597 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:56.195565    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh" event={"ID":"f526e30c-847e-4602-b4b1-f272c4e46c96","Type":"ContainerDied","Data":"f9911eb9656b24451cdb0967b31c0b817aced8e735c54a3d83e3d0fd2b93986e"}
Nov 02 08:05:56.404518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:56.404486    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"]
Nov 02 08:05:56.405062 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:05:56.405031    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda948aa04_1dd8_4822_88a4_2cb2ae7270b5.slice/crio-631f45bcce6c93295d654e40375ebe072a999e099cff22f571e2fbe8bd5fd1e7 WatchSource:0}: Error finding container 631f45bcce6c93295d654e40375ebe072a999e099cff22f571e2fbe8bd5fd1e7: Status 404 returned error can't find the container with id 631f45bcce6c93295d654e40375ebe072a999e099cff22f571e2fbe8bd5fd1e7
Nov 02 08:05:57.205291 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.205248    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg" event={"ID":"a948aa04-1dd8-4822-88a4-2cb2ae7270b5","Type":"ContainerStarted","Data":"5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633"}
Nov 02 08:05:57.205775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.205314    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg" event={"ID":"a948aa04-1dd8-4822-88a4-2cb2ae7270b5","Type":"ContainerStarted","Data":"631f45bcce6c93295d654e40375ebe072a999e099cff22f571e2fbe8bd5fd1e7"}
Nov 02 08:05:57.205775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.205523    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:05:57.251656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.251602    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg" podStartSLOduration=2.251586467 podStartE2EDuration="2.251586467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:05:57.248988532 +0000 UTC m=+2507.939315497" watchObservedRunningTime="2026-11-02 08:05:57.251586467 +0000 UTC m=+2507.941913431"
Nov 02 08:05:57.372082 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.372050    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:57.426748 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.426699    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-openshift-ca\") pod \"f526e30c-847e-4602-b4b1-f272c4e46c96\" (UID: \"f526e30c-847e-4602-b4b1-f272c4e46c96\") "
Nov 02 08:05:57.427075 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.427054    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f526e30c-847e-4602-b4b1-f272c4e46c96-kube-api-access-pspl8\" (UniqueName: \"kubernetes.io/projected/f526e30c-847e-4602-b4b1-f272c4e46c96-kube-api-access-pspl8\") pod \"f526e30c-847e-4602-b4b1-f272c4e46c96\" (UID: \"f526e30c-847e-4602-b4b1-f272c4e46c96\") "
Nov 02 08:05:57.427180 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.427139    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-openshift-ca" pod "f526e30c-847e-4602-b4b1-f272c4e46c96" (UID: "f526e30c-847e-4602-b4b1-f272c4e46c96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:05:57.427628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.427601    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-xks-ca\") pod \"f526e30c-847e-4602-b4b1-f272c4e46c96\" (UID: \"f526e30c-847e-4602-b4b1-f272c4e46c96\") "
Nov 02 08:05:57.428468 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.428440    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:57.430574 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.430541    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-xks-ca" pod "f526e30c-847e-4602-b4b1-f272c4e46c96" (UID: "f526e30c-847e-4602-b4b1-f272c4e46c96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:05:57.433988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.433961    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f526e30c-847e-4602-b4b1-f272c4e46c96-kube-api-access-pspl8" pod "f526e30c-847e-4602-b4b1-f272c4e46c96" (UID: "f526e30c-847e-4602-b4b1-f272c4e46c96"). InnerVolumeSpecName "kube-api-access-pspl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:05:57.529778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.529527    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pspl8\" (UniqueName: \"kubernetes.io/projected/f526e30c-847e-4602-b4b1-f272c4e46c96-kube-api-access-pspl8\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:57.529778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:57.529565    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f526e30c-847e-4602-b4b1-f272c4e46c96-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:05:58.215503 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:58.215468    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"
Nov 02 08:05:58.215503 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:58.215496    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh" event={"ID":"f526e30c-847e-4602-b4b1-f272c4e46c96","Type":"ContainerDied","Data":"aee379ce576ff5c84676ab6155c8329613993ae0317fe298b7c7169baa09924e"}
Nov 02 08:05:58.215980 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:58.215549    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aee379ce576ff5c84676ab6155c8329613993ae0317fe298b7c7169baa09924e"
Nov 02 08:05:59.782783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:05:59.782738    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"]
Nov 02 08:06:01.278666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.278626    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"]
Nov 02 08:06:01.279658 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.279631    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f526e30c-847e-4602-b4b1-f272c4e46c96" containerName="revoke-keys"
Nov 02 08:06:01.279658 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.279660    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f526e30c-847e-4602-b4b1-f272c4e46c96" containerName="revoke-keys"
Nov 02 08:06:01.279824 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.279789    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f526e30c-847e-4602-b4b1-f272c4e46c96" containerName="revoke-keys"
Nov 02 08:06:01.284210 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.284186    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:01.303711 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.303681    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"]
Nov 02 08:06:01.310126 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.310077    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:06:01.310288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.310231    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:06:01.310368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.310305    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:06:01.311096 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.311071    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:06:01.408485 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.408441    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1b0a3b4-db4zz\" (UID: \"2a71c5f7-5c2a-4735-848d-9706ff98f961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:01.408701 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.408561    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkwhr\" (UniqueName: \"kubernetes.io/projected/2a71c5f7-5c2a-4735-848d-9706ff98f961-kube-api-access-kkwhr\") pod \"maas-api-revoke-sub-b1b0a3b4-db4zz\" (UID: \"2a71c5f7-5c2a-4735-848d-9706ff98f961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:01.408839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.408817    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1b0a3b4-db4zz\" (UID: \"2a71c5f7-5c2a-4735-848d-9706ff98f961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:01.510068 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.510033    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1b0a3b4-db4zz\" (UID: \"2a71c5f7-5c2a-4735-848d-9706ff98f961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:01.510293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.510156    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1b0a3b4-db4zz\" (UID: \"2a71c5f7-5c2a-4735-848d-9706ff98f961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:01.510293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.510220    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkwhr\" (UniqueName: \"kubernetes.io/projected/2a71c5f7-5c2a-4735-848d-9706ff98f961-kube-api-access-kkwhr\") pod \"maas-api-revoke-sub-b1b0a3b4-db4zz\" (UID: \"2a71c5f7-5c2a-4735-848d-9706ff98f961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:01.510864 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.510841    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1b0a3b4-db4zz\" (UID: \"2a71c5f7-5c2a-4735-848d-9706ff98f961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:01.512718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.512686    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1b0a3b4-db4zz\" (UID: \"2a71c5f7-5c2a-4735-848d-9706ff98f961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:01.534139 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.534058    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkwhr\" (UniqueName: \"kubernetes.io/projected/2a71c5f7-5c2a-4735-848d-9706ff98f961-kube-api-access-kkwhr\") pod \"maas-api-revoke-sub-b1b0a3b4-db4zz\" (UID: \"2a71c5f7-5c2a-4735-848d-9706ff98f961\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:01.603038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.602999    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:01.917803 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:06:01.917772    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a71c5f7_5c2a_4735_848d_9706ff98f961.slice/crio-2864558f85a16ec20e02bf14cf0ca0dad7b3acf34ca0d5b4e554f154bed3ccc0 WatchSource:0}: Error finding container 2864558f85a16ec20e02bf14cf0ca0dad7b3acf34ca0d5b4e554f154bed3ccc0: Status 404 returned error can't find the container with id 2864558f85a16ec20e02bf14cf0ca0dad7b3acf34ca0d5b4e554f154bed3ccc0
Nov 02 08:06:01.918076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:01.918051    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"]
Nov 02 08:06:02.307223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:02.307185    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz" event={"ID":"2a71c5f7-5c2a-4735-848d-9706ff98f961","Type":"ContainerStarted","Data":"bb3741b5fbaf0d04d1141214c7276f9a5b8ce52e9a63967e099c80523c4b1e55"}
Nov 02 08:06:02.307691 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:02.307244    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz" event={"ID":"2a71c5f7-5c2a-4735-848d-9706ff98f961","Type":"ContainerStarted","Data":"2864558f85a16ec20e02bf14cf0ca0dad7b3acf34ca0d5b4e554f154bed3ccc0"}
Nov 02 08:06:02.349622 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:02.349569    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz" podStartSLOduration=1.349554199 podStartE2EDuration="1.349554199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:02.347444803 +0000 UTC m=+2513.037771767" watchObservedRunningTime="2026-11-02 08:06:02.349554199 +0000 UTC m=+2513.039881162"
Nov 02 08:06:03.317183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:03.317088    4898 generic.go:356] "Generic (PLEG): container finished" podID="2a71c5f7-5c2a-4735-848d-9706ff98f961" containerID="bb3741b5fbaf0d04d1141214c7276f9a5b8ce52e9a63967e099c80523c4b1e55" exitCode=0
Nov 02 08:06:03.317183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:03.317143    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz" event={"ID":"2a71c5f7-5c2a-4735-848d-9706ff98f961","Type":"ContainerDied","Data":"bb3741b5fbaf0d04d1141214c7276f9a5b8ce52e9a63967e099c80523c4b1e55"}
Nov 02 08:06:04.520237 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:04.520200    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:04.673173 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:04.673139    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a71c5f7-5c2a-4735-848d-9706ff98f961-kube-api-access-kkwhr\" (UniqueName: \"kubernetes.io/projected/2a71c5f7-5c2a-4735-848d-9706ff98f961-kube-api-access-kkwhr\") pod \"2a71c5f7-5c2a-4735-848d-9706ff98f961\" (UID: \"2a71c5f7-5c2a-4735-848d-9706ff98f961\") "
Nov 02 08:06:04.673398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:04.673210    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-openshift-ca\") pod \"2a71c5f7-5c2a-4735-848d-9706ff98f961\" (UID: \"2a71c5f7-5c2a-4735-848d-9706ff98f961\") "
Nov 02 08:06:04.673475 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:04.673425    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-xks-ca\") pod \"2a71c5f7-5c2a-4735-848d-9706ff98f961\" (UID: \"2a71c5f7-5c2a-4735-848d-9706ff98f961\") "
Nov 02 08:06:04.673752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:04.673716    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-openshift-ca" pod "2a71c5f7-5c2a-4735-848d-9706ff98f961" (UID: "2a71c5f7-5c2a-4735-848d-9706ff98f961"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:04.674215 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:04.674190    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:04.675666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:04.675628    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a71c5f7-5c2a-4735-848d-9706ff98f961-kube-api-access-kkwhr" pod "2a71c5f7-5c2a-4735-848d-9706ff98f961" (UID: "2a71c5f7-5c2a-4735-848d-9706ff98f961"). InnerVolumeSpecName "kube-api-access-kkwhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:04.675774 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:04.675635    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-xks-ca" pod "2a71c5f7-5c2a-4735-848d-9706ff98f961" (UID: "2a71c5f7-5c2a-4735-848d-9706ff98f961"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:06:04.775985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:04.775943    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a71c5f7-5c2a-4735-848d-9706ff98f961-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:04.775985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:04.775980    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkwhr\" (UniqueName: \"kubernetes.io/projected/2a71c5f7-5c2a-4735-848d-9706ff98f961-kube-api-access-kkwhr\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:04.783277 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:04.783236    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg" podUID="a948aa04-1dd8-4822-88a4-2cb2ae7270b5" containerName="payload-processing" containerID="cri-o://5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633" gracePeriod=25
Nov 02 08:06:05.050140 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.050111    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:06:05.181950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.181916    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-kube-api-access-rxrp6\" (UniqueName: \"kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-kube-api-access-rxrp6\") pod \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") "
Nov 02 08:06:05.182164 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.182006    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-routing-overlay-volume\") pod \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") "
Nov 02 08:06:05.182164 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.182135    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-plugins-config-volume\") pod \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") "
Nov 02 08:06:05.182302 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.182212    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-provider-credentials\") pod \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\" (UID: \"a948aa04-1dd8-4822-88a4-2cb2ae7270b5\") "
Nov 02 08:06:05.182460 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.182432    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-routing-overlay-volume" pod "a948aa04-1dd8-4822-88a4-2cb2ae7270b5" (UID: "a948aa04-1dd8-4822-88a4-2cb2ae7270b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:05.182565 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.182535    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-plugins-config-volume" pod "a948aa04-1dd8-4822-88a4-2cb2ae7270b5" (UID: "a948aa04-1dd8-4822-88a4-2cb2ae7270b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:05.183247 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.183219    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:05.183384 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.183251    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:05.184311 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.184282    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-kube-api-access-rxrp6" pod "a948aa04-1dd8-4822-88a4-2cb2ae7270b5" (UID: "a948aa04-1dd8-4822-88a4-2cb2ae7270b5"). InnerVolumeSpecName "kube-api-access-rxrp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:05.184615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.184596    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-provider-credentials" pod "a948aa04-1dd8-4822-88a4-2cb2ae7270b5" (UID: "a948aa04-1dd8-4822-88a4-2cb2ae7270b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:05.284779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.284691    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxrp6\" (UniqueName: \"kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-kube-api-access-rxrp6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:05.284779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.284724    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a948aa04-1dd8-4822-88a4-2cb2ae7270b5-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:05.344932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.344862    4898 generic.go:356] "Generic (PLEG): container finished" podID="a948aa04-1dd8-4822-88a4-2cb2ae7270b5" containerID="5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633" exitCode=0
Nov 02 08:06:05.345136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.344943    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"
Nov 02 08:06:05.345252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.345226    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg" event={"ID":"a948aa04-1dd8-4822-88a4-2cb2ae7270b5","Type":"ContainerDied","Data":"5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633"}
Nov 02 08:06:05.345333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.345286    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg" event={"ID":"a948aa04-1dd8-4822-88a4-2cb2ae7270b5","Type":"ContainerDied","Data":"631f45bcce6c93295d654e40375ebe072a999e099cff22f571e2fbe8bd5fd1e7"}
Nov 02 08:06:05.345333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.345319    4898 scope.go:122] "RemoveContainer" containerID="5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633"
Nov 02 08:06:05.348438 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.348401    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz" event={"ID":"2a71c5f7-5c2a-4735-848d-9706ff98f961","Type":"ContainerDied","Data":"2864558f85a16ec20e02bf14cf0ca0dad7b3acf34ca0d5b4e554f154bed3ccc0"}
Nov 02 08:06:05.348544 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.348454    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2864558f85a16ec20e02bf14cf0ca0dad7b3acf34ca0d5b4e554f154bed3ccc0"
Nov 02 08:06:05.348544 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.348515    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"
Nov 02 08:06:05.366150 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.366121    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"]
Nov 02 08:06:05.372194 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.369444    4898 scope.go:122] "RemoveContainer" containerID="5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633"
Nov 02 08:06:05.375184 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:06:05.375113    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633\": container with ID starting with 5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633 not found: ID does not exist" containerID="5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633"
Nov 02 08:06:05.375184 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.375156    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633"} err="failed to get container status \"5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633\": rpc error: code = NotFound desc = could not find container \"5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633\": container with ID starting with 5b54d960dc5f53b4d8dcdd18bcbd2054ee5a4cfc9df7f1e8462d1bb161cdb633 not found: ID does not exist"
Nov 02 08:06:05.410531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.410481    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"]
Nov 02 08:06:05.416481 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.416450    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-64f4zwrdg"]
Nov 02 08:06:05.468168 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:05.468120    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a948aa04-1dd8-4822-88a4-2cb2ae7270b5" path="/var/lib/kubelet/pods/a948aa04-1dd8-4822-88a4-2cb2ae7270b5/volumes"
Nov 02 08:06:10.371912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.366948    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh" podUID="e9ab319e-8c32-4ed7-8953-cb248bea36e6" containerName="payload-processing" containerID="cri-o://1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0" gracePeriod=25
Nov 02 08:06:10.526646 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:06:10.526595    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ab319e_8c32_4ed7_8953_cb248bea36e6.slice/crio-conmon-1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ab319e_8c32_4ed7_8953_cb248bea36e6.slice/crio-1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 08:06:10.706728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.706697    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:06:10.859311 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.859278    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-routing-overlay-volume\") pod \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") "
Nov 02 08:06:10.859541 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.859420    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-provider-credentials\") pod \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") "
Nov 02 08:06:10.859541 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.859455    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-kube-api-access-g659w\" (UniqueName: \"kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-kube-api-access-g659w\") pod \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") "
Nov 02 08:06:10.859732 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.859693    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-routing-overlay-volume" pod "e9ab319e-8c32-4ed7-8953-cb248bea36e6" (UID: "e9ab319e-8c32-4ed7-8953-cb248bea36e6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:10.859732 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.859717    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-plugins-config-volume\") pod \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\" (UID: \"e9ab319e-8c32-4ed7-8953-cb248bea36e6\") "
Nov 02 08:06:10.860130 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.860100    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-plugins-config-volume" pod "e9ab319e-8c32-4ed7-8953-cb248bea36e6" (UID: "e9ab319e-8c32-4ed7-8953-cb248bea36e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:10.860517 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.860494    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:10.860517 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.860517    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9ab319e-8c32-4ed7-8953-cb248bea36e6-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:10.861744 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.861719    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-provider-credentials" pod "e9ab319e-8c32-4ed7-8953-cb248bea36e6" (UID: "e9ab319e-8c32-4ed7-8953-cb248bea36e6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:10.861822 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.861754    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-kube-api-access-g659w" pod "e9ab319e-8c32-4ed7-8953-cb248bea36e6" (UID: "e9ab319e-8c32-4ed7-8953-cb248bea36e6"). InnerVolumeSpecName "kube-api-access-g659w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:10.961992 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.961859    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:10.961992 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:10.961940    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g659w\" (UniqueName: \"kubernetes.io/projected/e9ab319e-8c32-4ed7-8953-cb248bea36e6-kube-api-access-g659w\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:11.473098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:11.473048    4898 generic.go:356] "Generic (PLEG): container finished" podID="e9ab319e-8c32-4ed7-8953-cb248bea36e6" containerID="1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0" exitCode=0
Nov 02 08:06:11.473541 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:11.473177    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"
Nov 02 08:06:11.480644 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:11.480599    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh" event={"ID":"e9ab319e-8c32-4ed7-8953-cb248bea36e6","Type":"ContainerDied","Data":"1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0"}
Nov 02 08:06:11.480804 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:11.480660    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh" event={"ID":"e9ab319e-8c32-4ed7-8953-cb248bea36e6","Type":"ContainerDied","Data":"9729e68c0bcaad25744181eddecb07f1711266a188c152f1195a20fb43f5d219"}
Nov 02 08:06:11.480804 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:11.480699    4898 scope.go:122] "RemoveContainer" containerID="1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0"
Nov 02 08:06:11.497081 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:11.496972    4898 scope.go:122] "RemoveContainer" containerID="1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0"
Nov 02 08:06:11.497690 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:06:11.497665    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0\": container with ID starting with 1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0 not found: ID does not exist" containerID="1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0"
Nov 02 08:06:11.497783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:11.497702    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0"} err="failed to get container status \"1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0\": rpc error: code = NotFound desc = could not find container \"1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0\": container with ID starting with 1544f1207e1ebeb0d6ca6c682705e3789a4a6697d5cfe2b389c5a2e6433de7c0 not found: ID does not exist"
Nov 02 08:06:11.576106 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:11.576069    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"]
Nov 02 08:06:11.589508 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:11.589472    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bb4907/payload-processing-external-model-e2e-extmod-b-bb4907-8b89kgjrh"]
Nov 02 08:06:12.658713 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:12.658678    4898 scope.go:122] "RemoveContainer" containerID="3b9f1be42dbdc54b713d5ee077690ebdbfe51b5b1b96ddb89887fa3b653a6ab6"
Nov 02 08:06:12.669699 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:12.669665    4898 scope.go:122] "RemoveContainer" containerID="38c19ac316a0d5fb539fb92f0a18bfafe59f51758cf00e3fa54e7c14fcbf21b7"
Nov 02 08:06:12.680814 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:12.680790    4898 scope.go:122] "RemoveContainer" containerID="13312acd61b87e2b179500a2042e41779470285ecd58481cdd7d033bcb9248c2"
Nov 02 08:06:12.691799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:12.691776    4898 scope.go:122] "RemoveContainer" containerID="5d43cdc91bc558692e8ce405c9811bf874007404526fc2249d892639d4b5e48d"
Nov 02 08:06:13.473110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.473072    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ab319e-8c32-4ed7-8953-cb248bea36e6" path="/var/lib/kubelet/pods/e9ab319e-8c32-4ed7-8953-cb248bea36e6/volumes"
Nov 02 08:06:13.473632 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.473600    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"]
Nov 02 08:06:13.474704 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.474684    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a948aa04-1dd8-4822-88a4-2cb2ae7270b5" containerName="payload-processing"
Nov 02 08:06:13.474792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.474707    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a948aa04-1dd8-4822-88a4-2cb2ae7270b5" containerName="payload-processing"
Nov 02 08:06:13.474792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.474738    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2a71c5f7-5c2a-4735-848d-9706ff98f961" containerName="revoke-keys"
Nov 02 08:06:13.474792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.474746    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a71c5f7-5c2a-4735-848d-9706ff98f961" containerName="revoke-keys"
Nov 02 08:06:13.474792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.474770    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e9ab319e-8c32-4ed7-8953-cb248bea36e6" containerName="payload-processing"
Nov 02 08:06:13.474792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.474778    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ab319e-8c32-4ed7-8953-cb248bea36e6" containerName="payload-processing"
Nov 02 08:06:13.475006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.474993    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a71c5f7-5c2a-4735-848d-9706ff98f961" containerName="revoke-keys"
Nov 02 08:06:13.475046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.475016    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ab319e-8c32-4ed7-8953-cb248bea36e6" containerName="payload-processing"
Nov 02 08:06:13.475046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.475039    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a948aa04-1dd8-4822-88a4-2cb2ae7270b5" containerName="payload-processing"
Nov 02 08:06:13.482918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.482435    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:13.496666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.495538    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"]
Nov 02 08:06:13.519413 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.507785    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:06:13.519413 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.507980    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:06:13.519413 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.508602    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:06:13.535713 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.526866    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:06:13.549920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.547323    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289f0b6d-pwbgs\" (UID: \"f32ebe67-ad83-407b-b75b-6fc41e53b807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:13.549920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.547419    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kg9\" (UniqueName: \"kubernetes.io/projected/f32ebe67-ad83-407b-b75b-6fc41e53b807-kube-api-access-55kg9\") pod \"maas-api-revoke-sub-289f0b6d-pwbgs\" (UID: \"f32ebe67-ad83-407b-b75b-6fc41e53b807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:13.549920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.547532    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289f0b6d-pwbgs\" (UID: \"f32ebe67-ad83-407b-b75b-6fc41e53b807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:13.692412 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.692358    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289f0b6d-pwbgs\" (UID: \"f32ebe67-ad83-407b-b75b-6fc41e53b807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:13.696911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.696847    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289f0b6d-pwbgs\" (UID: \"f32ebe67-ad83-407b-b75b-6fc41e53b807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:13.698096 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.695883    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289f0b6d-pwbgs\" (UID: \"f32ebe67-ad83-407b-b75b-6fc41e53b807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:13.698904 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.698371    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kg9\" (UniqueName: \"kubernetes.io/projected/f32ebe67-ad83-407b-b75b-6fc41e53b807-kube-api-access-55kg9\") pod \"maas-api-revoke-sub-289f0b6d-pwbgs\" (UID: \"f32ebe67-ad83-407b-b75b-6fc41e53b807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:13.722472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.722438    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289f0b6d-pwbgs\" (UID: \"f32ebe67-ad83-407b-b75b-6fc41e53b807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:13.728604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.728480    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kg9\" (UniqueName: \"kubernetes.io/projected/f32ebe67-ad83-407b-b75b-6fc41e53b807-kube-api-access-55kg9\") pod \"maas-api-revoke-sub-289f0b6d-pwbgs\" (UID: \"f32ebe67-ad83-407b-b75b-6fc41e53b807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:13.872611 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.872575    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"]
Nov 02 08:06:13.877612 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.877587    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:13.894204 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.894170    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"routing-overlay\""
Nov 02 08:06:13.894393 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.894272    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2dde84\""
Nov 02 08:06:13.894393 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.894276    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"openshift-service-ca.crt\""
Nov 02 08:06:13.894393 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.894340    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"kube-root-ca.crt\""
Nov 02 08:06:13.896380 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.896357    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"]
Nov 02 08:06:13.909478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.909448    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:13.911301 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.911273    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-e2e--1a3816b7-dockercfg-4nccn\""
Nov 02 08:06:13.911444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:13.911313    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"agc-cred-74b36e68d5bd\""
Nov 02 08:06:14.005335 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.005297    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkd5b\" (UniqueName: \"kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-kube-api-access-xkd5b\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.005483 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.005427    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.005483 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.005463    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.005843 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.005817    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.107576 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.107532    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.107773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.107613    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkd5b\" (UniqueName: \"kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-kube-api-access-xkd5b\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.107773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.107697    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.107773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.107723    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.108297 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.108273    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.108625 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.108597    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.110374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.110352    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.131757 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.131716    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkd5b\" (UniqueName: \"kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-kube-api-access-xkd5b\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.193559 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.193522    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:14.225197 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.225146    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"]
Nov 02 08:06:14.229006 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:06:14.228968    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32ebe67_ad83_407b_b75b_6fc41e53b807.slice/crio-2a095c4f629417cf397dae614b61a6c0418b51e124d0fac4c5013b1ee06c7068 WatchSource:0}: Error finding container 2a095c4f629417cf397dae614b61a6c0418b51e124d0fac4c5013b1ee06c7068: Status 404 returned error can't find the container with id 2a095c4f629417cf397dae614b61a6c0418b51e124d0fac4c5013b1ee06c7068
Nov 02 08:06:14.519300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.519210    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"]
Nov 02 08:06:14.524496 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:06:14.524467    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36767b70_b986_47ff_9e6b_25c2342e6898.slice/crio-a4f8efbd53706fe212298090b2269b5fbb737f5124db76211fe686ae9e89d186 WatchSource:0}: Error finding container a4f8efbd53706fe212298090b2269b5fbb737f5124db76211fe686ae9e89d186: Status 404 returned error can't find the container with id a4f8efbd53706fe212298090b2269b5fbb737f5124db76211fe686ae9e89d186
Nov 02 08:06:14.534407 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:14.534375    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs" event={"ID":"f32ebe67-ad83-407b-b75b-6fc41e53b807","Type":"ContainerStarted","Data":"2a095c4f629417cf397dae614b61a6c0418b51e124d0fac4c5013b1ee06c7068"}
Nov 02 08:06:14.991272 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:06:14.991227    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:06:15.546069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:15.545982    4898 generic.go:356] "Generic (PLEG): container finished" podID="f32ebe67-ad83-407b-b75b-6fc41e53b807" containerID="3be2d174a658da07e6eaba3d7c1e5d849e3d92deaf31e5d7018d9f5008c86c51" exitCode=0
Nov 02 08:06:15.546268 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:15.546067    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs" event={"ID":"f32ebe67-ad83-407b-b75b-6fc41e53b807","Type":"ContainerDied","Data":"3be2d174a658da07e6eaba3d7c1e5d849e3d92deaf31e5d7018d9f5008c86c51"}
Nov 02 08:06:15.547788 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:15.547760    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d" event={"ID":"36767b70-b986-47ff-9e6b-25c2342e6898","Type":"ContainerStarted","Data":"0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e"}
Nov 02 08:06:15.547921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:15.547803    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d" event={"ID":"36767b70-b986-47ff-9e6b-25c2342e6898","Type":"ContainerStarted","Data":"a4f8efbd53706fe212298090b2269b5fbb737f5124db76211fe686ae9e89d186"}
Nov 02 08:06:15.547966 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:15.547923    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:15.631498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:15.631445    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d" podStartSLOduration=2.631431401 podStartE2EDuration="2.631431401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:15.627028271 +0000 UTC m=+2526.317355262" watchObservedRunningTime="2026-11-02 08:06:15.631431401 +0000 UTC m=+2526.321758365"
Nov 02 08:06:16.733225 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:16.733195    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:16.853679 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:16.853581    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-openshift-ca\") pod \"f32ebe67-ad83-407b-b75b-6fc41e53b807\" (UID: \"f32ebe67-ad83-407b-b75b-6fc41e53b807\") "
Nov 02 08:06:16.853679 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:16.853633    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32ebe67-ad83-407b-b75b-6fc41e53b807-kube-api-access-55kg9\" (UniqueName: \"kubernetes.io/projected/f32ebe67-ad83-407b-b75b-6fc41e53b807-kube-api-access-55kg9\") pod \"f32ebe67-ad83-407b-b75b-6fc41e53b807\" (UID: \"f32ebe67-ad83-407b-b75b-6fc41e53b807\") "
Nov 02 08:06:16.853967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:16.853682    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-xks-ca\") pod \"f32ebe67-ad83-407b-b75b-6fc41e53b807\" (UID: \"f32ebe67-ad83-407b-b75b-6fc41e53b807\") "
Nov 02 08:06:16.854054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:16.854024    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-openshift-ca" pod "f32ebe67-ad83-407b-b75b-6fc41e53b807" (UID: "f32ebe67-ad83-407b-b75b-6fc41e53b807"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:16.854222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:16.854206    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:16.855975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:16.855947    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32ebe67-ad83-407b-b75b-6fc41e53b807-kube-api-access-55kg9" pod "f32ebe67-ad83-407b-b75b-6fc41e53b807" (UID: "f32ebe67-ad83-407b-b75b-6fc41e53b807"). InnerVolumeSpecName "kube-api-access-55kg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:16.856111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:16.856026    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-xks-ca" pod "f32ebe67-ad83-407b-b75b-6fc41e53b807" (UID: "f32ebe67-ad83-407b-b75b-6fc41e53b807"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:06:16.955722 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:16.955681    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55kg9\" (UniqueName: \"kubernetes.io/projected/f32ebe67-ad83-407b-b75b-6fc41e53b807-kube-api-access-55kg9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:16.955722 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:16.955722    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32ebe67-ad83-407b-b75b-6fc41e53b807-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:17.573091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:17.573046    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs" event={"ID":"f32ebe67-ad83-407b-b75b-6fc41e53b807","Type":"ContainerDied","Data":"2a095c4f629417cf397dae614b61a6c0418b51e124d0fac4c5013b1ee06c7068"}
Nov 02 08:06:17.573279 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:17.573117    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a095c4f629417cf397dae614b61a6c0418b51e124d0fac4c5013b1ee06c7068"
Nov 02 08:06:17.573279 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:17.573191    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"
Nov 02 08:06:24.822120 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:24.822084    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"]
Nov 02 08:06:24.825030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:24.825003    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:27.299438 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.299389    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"]
Nov 02 08:06:27.300580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.300551    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f32ebe67-ad83-407b-b75b-6fc41e53b807" containerName="revoke-keys"
Nov 02 08:06:27.300580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.300579    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32ebe67-ad83-407b-b75b-6fc41e53b807" containerName="revoke-keys"
Nov 02 08:06:27.300820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.300807    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32ebe67-ad83-407b-b75b-6fc41e53b807" containerName="revoke-keys"
Nov 02 08:06:27.306425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.306400    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:27.326984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.326950    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"]
Nov 02 08:06:27.331566 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.331538    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:06:27.331749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.331607    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:06:27.331840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.331670    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:06:27.333946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.333866    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:06:27.359546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.359503    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e35aec7-bvk7f\" (UID: \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:27.360492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.360450    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28lw4\" (UniqueName: \"kubernetes.io/projected/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-kube-api-access-28lw4\") pod \"maas-api-revoke-sub-7e35aec7-bvk7f\" (UID: \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:27.360673 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.360578    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e35aec7-bvk7f\" (UID: \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:27.461970 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.461913    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lw4\" (UniqueName: \"kubernetes.io/projected/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-kube-api-access-28lw4\") pod \"maas-api-revoke-sub-7e35aec7-bvk7f\" (UID: \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:27.462182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.462027    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e35aec7-bvk7f\" (UID: \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:27.462182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.462174    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e35aec7-bvk7f\" (UID: \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:27.463098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.463065    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e35aec7-bvk7f\" (UID: \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:27.465744 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.465712    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e35aec7-bvk7f\" (UID: \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:27.486330 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.486297    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28lw4\" (UniqueName: \"kubernetes.io/projected/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-kube-api-access-28lw4\") pod \"maas-api-revoke-sub-7e35aec7-bvk7f\" (UID: \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:27.625231 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.625145    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:27.953755 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:06:27.953717    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9edc2e6_2c5d_4cb8_b6d4_6595e7b472af.slice/crio-039d468ded26f2577ec111a63f25431dd03563342482de7ba24323ad862fdfcc WatchSource:0}: Error finding container 039d468ded26f2577ec111a63f25431dd03563342482de7ba24323ad862fdfcc: Status 404 returned error can't find the container with id 039d468ded26f2577ec111a63f25431dd03563342482de7ba24323ad862fdfcc
Nov 02 08:06:27.955595 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:27.955154    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"]
Nov 02 08:06:28.690733 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:28.690693    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f" event={"ID":"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af","Type":"ContainerStarted","Data":"2179f10f52358457e5e1de3537895af4498b948f6423caaac5a6e696c24aa99c"}
Nov 02 08:06:28.691288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:28.690757    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f" event={"ID":"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af","Type":"ContainerStarted","Data":"039d468ded26f2577ec111a63f25431dd03563342482de7ba24323ad862fdfcc"}
Nov 02 08:06:29.701367 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:29.701324    4898 generic.go:356] "Generic (PLEG): container finished" podID="f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af" containerID="2179f10f52358457e5e1de3537895af4498b948f6423caaac5a6e696c24aa99c" exitCode=0
Nov 02 08:06:29.701810 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:29.701416    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f" event={"ID":"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af","Type":"ContainerDied","Data":"2179f10f52358457e5e1de3537895af4498b948f6423caaac5a6e696c24aa99c"}
Nov 02 08:06:29.822786 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:29.822732    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d" podUID="36767b70-b986-47ff-9e6b-25c2342e6898" containerName="payload-processing" containerID="cri-o://0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e" gracePeriod=25
Nov 02 08:06:30.397323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.397291    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:30.462331 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.462297    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-plugins-config-volume\") pod \"36767b70-b986-47ff-9e6b-25c2342e6898\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") "
Nov 02 08:06:30.462533 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.462429    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-routing-overlay-volume\") pod \"36767b70-b986-47ff-9e6b-25c2342e6898\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") "
Nov 02 08:06:30.462533 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.462503    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-provider-credentials\" (UniqueName: \"kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-provider-credentials\") pod \"36767b70-b986-47ff-9e6b-25c2342e6898\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") "
Nov 02 08:06:30.462710 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.462688    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-kube-api-access-xkd5b\" (UniqueName: \"kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-kube-api-access-xkd5b\") pod \"36767b70-b986-47ff-9e6b-25c2342e6898\" (UID: \"36767b70-b986-47ff-9e6b-25c2342e6898\") "
Nov 02 08:06:30.462838 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.462813    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-routing-overlay-volume" pod "36767b70-b986-47ff-9e6b-25c2342e6898" (UID: "36767b70-b986-47ff-9e6b-25c2342e6898"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:30.462929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.462812    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-plugins-config-volume" pod "36767b70-b986-47ff-9e6b-25c2342e6898" (UID: "36767b70-b986-47ff-9e6b-25c2342e6898"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:30.463297 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.463270    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:30.463297 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.463298    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36767b70-b986-47ff-9e6b-25c2342e6898-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:30.464906 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.464861    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-provider-credentials" pod "36767b70-b986-47ff-9e6b-25c2342e6898" (UID: "36767b70-b986-47ff-9e6b-25c2342e6898"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:30.465099 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.465079    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-kube-api-access-xkd5b" pod "36767b70-b986-47ff-9e6b-25c2342e6898" (UID: "36767b70-b986-47ff-9e6b-25c2342e6898"). InnerVolumeSpecName "kube-api-access-xkd5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:30.564998 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.564957    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkd5b\" (UniqueName: \"kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-kube-api-access-xkd5b\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:30.564998 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.564994    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36767b70-b986-47ff-9e6b-25c2342e6898-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:30.712588 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.712550    4898 generic.go:356] "Generic (PLEG): container finished" podID="36767b70-b986-47ff-9e6b-25c2342e6898" containerID="0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e" exitCode=0
Nov 02 08:06:30.713165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.713048    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"
Nov 02 08:06:30.716237 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.715972    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d" event={"ID":"36767b70-b986-47ff-9e6b-25c2342e6898","Type":"ContainerDied","Data":"0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e"}
Nov 02 08:06:30.716237 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.716036    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d" event={"ID":"36767b70-b986-47ff-9e6b-25c2342e6898","Type":"ContainerDied","Data":"a4f8efbd53706fe212298090b2269b5fbb737f5124db76211fe686ae9e89d186"}
Nov 02 08:06:30.716237 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.716068    4898 scope.go:122] "RemoveContainer" containerID="0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e"
Nov 02 08:06:30.814054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.813389    4898 scope.go:122] "RemoveContainer" containerID="0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e"
Nov 02 08:06:30.814054 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:06:30.813813    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e\": container with ID starting with 0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e not found: ID does not exist" containerID="0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e"
Nov 02 08:06:30.814054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.813852    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e"} err="failed to get container status \"0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e\": rpc error: code = NotFound desc = could not find container \"0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e\": container with ID starting with 0647b1fa2803f87b431dde72c1fd647852e0f9ca87c4bb8330b9b8f60eb45f5e not found: ID does not exist"
Nov 02 08:06:30.850380 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.847144    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"]
Nov 02 08:06:30.854842 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:30.854806    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-5587dv98d"]
Nov 02 08:06:31.025263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.025236    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:31.077566 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.077526    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-openshift-ca\") pod \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\" (UID: \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\") "
Nov 02 08:06:31.077764 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.077651    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-kube-api-access-28lw4\" (UniqueName: \"kubernetes.io/projected/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-kube-api-access-28lw4\") pod \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\" (UID: \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\") "
Nov 02 08:06:31.077840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.077820    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-xks-ca\") pod \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\" (UID: \"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af\") "
Nov 02 08:06:31.078000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.077970    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-openshift-ca" pod "f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af" (UID: "f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:31.078970 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.078871    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:31.083955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.080628    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-kube-api-access-28lw4" pod "f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af" (UID: "f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af"). InnerVolumeSpecName "kube-api-access-28lw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:31.083955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.080675    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-xks-ca" pod "f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af" (UID: "f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:06:31.180769 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.180728    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:31.180769 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.180761    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28lw4\" (UniqueName: \"kubernetes.io/projected/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af-kube-api-access-28lw4\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:31.466931 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.466858    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36767b70-b986-47ff-9e6b-25c2342e6898" path="/var/lib/kubelet/pods/36767b70-b986-47ff-9e6b-25c2342e6898/volumes"
Nov 02 08:06:31.723843 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.723754    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"
Nov 02 08:06:31.723843 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.723791    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f" event={"ID":"f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af","Type":"ContainerDied","Data":"039d468ded26f2577ec111a63f25431dd03563342482de7ba24323ad862fdfcc"}
Nov 02 08:06:31.724316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:31.723879    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="039d468ded26f2577ec111a63f25431dd03563342482de7ba24323ad862fdfcc"
Nov 02 08:06:39.283905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.283857    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"]
Nov 02 08:06:39.284812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.284786    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af" containerName="revoke-keys"
Nov 02 08:06:39.284812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.284812    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af" containerName="revoke-keys"
Nov 02 08:06:39.285043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.284901    4898 cpu_manager.go:414] "containerMap: removing container" podUID="36767b70-b986-47ff-9e6b-25c2342e6898" containerName="payload-processing"
Nov 02 08:06:39.285043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.284911    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36767b70-b986-47ff-9e6b-25c2342e6898" containerName="payload-processing"
Nov 02 08:06:39.285155 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.285093    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="36767b70-b986-47ff-9e6b-25c2342e6898" containerName="payload-processing"
Nov 02 08:06:39.285155 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.285115    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af" containerName="revoke-keys"
Nov 02 08:06:39.288989 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.288966    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.306975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.306946    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"]
Nov 02 08:06:39.444931 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.444881    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.445144 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.445068    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqhn\" (UniqueName: \"kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-kube-api-access-fhqhn\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.445226 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.445160    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.445226 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.445208    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.454748 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.454716    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"]
Nov 02 08:06:39.461304 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.461279    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:39.478423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.478390    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"]
Nov 02 08:06:39.479020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.478998    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:06:39.479293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.479272    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:06:39.479372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.479305    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:06:39.479372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.479319    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:06:39.547042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.546952    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.547205 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.547049    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.547205 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.547122    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.547323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.547258    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-297e7586-p6qvl\" (UID: \"e88dc09f-e44c-4b86-9f28-3b27722ac761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:39.547416 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.547385    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfkgr\" (UniqueName: \"kubernetes.io/projected/e88dc09f-e44c-4b86-9f28-3b27722ac761-kube-api-access-dfkgr\") pod \"maas-api-revoke-sub-297e7586-p6qvl\" (UID: \"e88dc09f-e44c-4b86-9f28-3b27722ac761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:39.547488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.547461    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqhn\" (UniqueName: \"kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-kube-api-access-fhqhn\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.547549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.547518    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-297e7586-p6qvl\" (UID: \"e88dc09f-e44c-4b86-9f28-3b27722ac761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:39.547847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.547825    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.547847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.547840    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.549705 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.549685    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.572986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.572948    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqhn\" (UniqueName: \"kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-kube-api-access-fhqhn\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.611912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.608429    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:39.764987 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.736033    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-297e7586-p6qvl\" (UID: \"e88dc09f-e44c-4b86-9f28-3b27722ac761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:39.764987 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.738242    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-297e7586-p6qvl\" (UID: \"e88dc09f-e44c-4b86-9f28-3b27722ac761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:39.764987 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.738369    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfkgr\" (UniqueName: \"kubernetes.io/projected/e88dc09f-e44c-4b86-9f28-3b27722ac761-kube-api-access-dfkgr\") pod \"maas-api-revoke-sub-297e7586-p6qvl\" (UID: \"e88dc09f-e44c-4b86-9f28-3b27722ac761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:39.764987 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.739935    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-297e7586-p6qvl\" (UID: \"e88dc09f-e44c-4b86-9f28-3b27722ac761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:39.789912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.782485    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-297e7586-p6qvl\" (UID: \"e88dc09f-e44c-4b86-9f28-3b27722ac761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:39.800489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:39.798171    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfkgr\" (UniqueName: \"kubernetes.io/projected/e88dc09f-e44c-4b86-9f28-3b27722ac761-kube-api-access-dfkgr\") pod \"maas-api-revoke-sub-297e7586-p6qvl\" (UID: \"e88dc09f-e44c-4b86-9f28-3b27722ac761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:40.078310 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:40.078233    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:40.107184 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:40.107156    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"]
Nov 02 08:06:40.107315 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:06:40.107171    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b48a31_4253_4e82_96ff_38685f281f68.slice/crio-cc312547c025f68f54d86cbb6f90e8743c51f6f89d791a22b1af0fe81e76248e WatchSource:0}: Error finding container cc312547c025f68f54d86cbb6f90e8743c51f6f89d791a22b1af0fe81e76248e: Status 404 returned error can't find the container with id cc312547c025f68f54d86cbb6f90e8743c51f6f89d791a22b1af0fe81e76248e
Nov 02 08:06:40.391300 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:06:40.391267    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88dc09f_e44c_4b86_9f28_3b27722ac761.slice/crio-28573de0883019fd9a3b439c8f941217f3c226ca037409f3daecdb50a33294c7 WatchSource:0}: Error finding container 28573de0883019fd9a3b439c8f941217f3c226ca037409f3daecdb50a33294c7: Status 404 returned error can't find the container with id 28573de0883019fd9a3b439c8f941217f3c226ca037409f3daecdb50a33294c7
Nov 02 08:06:40.392885 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:40.392862    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"]
Nov 02 08:06:40.959423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:40.956226    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw" event={"ID":"d5b48a31-4253-4e82-96ff-38685f281f68","Type":"ContainerStarted","Data":"88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f"}
Nov 02 08:06:40.959423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:40.956347    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw" event={"ID":"d5b48a31-4253-4e82-96ff-38685f281f68","Type":"ContainerStarted","Data":"cc312547c025f68f54d86cbb6f90e8743c51f6f89d791a22b1af0fe81e76248e"}
Nov 02 08:06:40.966107 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:40.966072    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:40.993913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:40.990112    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl" event={"ID":"e88dc09f-e44c-4b86-9f28-3b27722ac761","Type":"ContainerStarted","Data":"d05a22630e09fa9d44ff1d14e04699c5dc12ff354bac0061706691701beb9800"}
Nov 02 08:06:40.993913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:40.990180    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl" event={"ID":"e88dc09f-e44c-4b86-9f28-3b27722ac761","Type":"ContainerStarted","Data":"28573de0883019fd9a3b439c8f941217f3c226ca037409f3daecdb50a33294c7"}
Nov 02 08:06:41.020946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:41.020805    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw" podStartSLOduration=2.020783868 podStartE2EDuration="2.020783868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:41.015541538 +0000 UTC m=+2551.705868502" watchObservedRunningTime="2026-11-02 08:06:41.020783868 +0000 UTC m=+2551.711110833"
Nov 02 08:06:41.067538 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:41.067468    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl" podStartSLOduration=2.06744762 podStartE2EDuration="2.06744762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:41.061408474 +0000 UTC m=+2551.751735458" watchObservedRunningTime="2026-11-02 08:06:41.06744762 +0000 UTC m=+2551.757774584"
Nov 02 08:06:42.003683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:42.003642    4898 generic.go:356] "Generic (PLEG): container finished" podID="e88dc09f-e44c-4b86-9f28-3b27722ac761" containerID="d05a22630e09fa9d44ff1d14e04699c5dc12ff354bac0061706691701beb9800" exitCode=0
Nov 02 08:06:42.004163 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:42.003725    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl" event={"ID":"e88dc09f-e44c-4b86-9f28-3b27722ac761","Type":"ContainerDied","Data":"d05a22630e09fa9d44ff1d14e04699c5dc12ff354bac0061706691701beb9800"}
Nov 02 08:06:43.293965 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:43.293151    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:43.421606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:43.421574    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-openshift-ca\") pod \"e88dc09f-e44c-4b86-9f28-3b27722ac761\" (UID: \"e88dc09f-e44c-4b86-9f28-3b27722ac761\") "
Nov 02 08:06:43.421788 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:43.421624    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88dc09f-e44c-4b86-9f28-3b27722ac761-kube-api-access-dfkgr\" (UniqueName: \"kubernetes.io/projected/e88dc09f-e44c-4b86-9f28-3b27722ac761-kube-api-access-dfkgr\") pod \"e88dc09f-e44c-4b86-9f28-3b27722ac761\" (UID: \"e88dc09f-e44c-4b86-9f28-3b27722ac761\") "
Nov 02 08:06:43.421861 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:43.421842    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-xks-ca\") pod \"e88dc09f-e44c-4b86-9f28-3b27722ac761\" (UID: \"e88dc09f-e44c-4b86-9f28-3b27722ac761\") "
Nov 02 08:06:43.422000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:43.421977    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-openshift-ca" pod "e88dc09f-e44c-4b86-9f28-3b27722ac761" (UID: "e88dc09f-e44c-4b86-9f28-3b27722ac761"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:43.422469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:43.422444    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:43.423918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:43.423857    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-xks-ca" pod "e88dc09f-e44c-4b86-9f28-3b27722ac761" (UID: "e88dc09f-e44c-4b86-9f28-3b27722ac761"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:06:43.424012 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:43.423964    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88dc09f-e44c-4b86-9f28-3b27722ac761-kube-api-access-dfkgr" pod "e88dc09f-e44c-4b86-9f28-3b27722ac761" (UID: "e88dc09f-e44c-4b86-9f28-3b27722ac761"). InnerVolumeSpecName "kube-api-access-dfkgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:43.541949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:43.541172    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfkgr\" (UniqueName: \"kubernetes.io/projected/e88dc09f-e44c-4b86-9f28-3b27722ac761-kube-api-access-dfkgr\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:43.541949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:43.541209    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88dc09f-e44c-4b86-9f28-3b27722ac761-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:44.026885 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:44.026843    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl" event={"ID":"e88dc09f-e44c-4b86-9f28-3b27722ac761","Type":"ContainerDied","Data":"28573de0883019fd9a3b439c8f941217f3c226ca037409f3daecdb50a33294c7"}
Nov 02 08:06:44.026885 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:44.026875    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"
Nov 02 08:06:44.027117 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:44.026910    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28573de0883019fd9a3b439c8f941217f3c226ca037409f3daecdb50a33294c7"
Nov 02 08:06:44.993125 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:06:44.993085    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:06:45.215446 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.215405    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"]
Nov 02 08:06:45.216227 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.216209    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e88dc09f-e44c-4b86-9f28-3b27722ac761" containerName="revoke-keys"
Nov 02 08:06:45.216280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.216227    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88dc09f-e44c-4b86-9f28-3b27722ac761" containerName="revoke-keys"
Nov 02 08:06:45.216349 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.216339    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88dc09f-e44c-4b86-9f28-3b27722ac761" containerName="revoke-keys"
Nov 02 08:06:45.220472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.220451    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.239356 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.239326    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"]
Nov 02 08:06:45.246728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.246641    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-736f7c\""
Nov 02 08:06:45.246728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.246644    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"kube-root-ca.crt\""
Nov 02 08:06:45.248072 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.248048    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"e2e-cfl-4704f3-provider-cred\""
Nov 02 08:06:45.248214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.248078    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"openshift-service-ca.crt\""
Nov 02 08:06:45.248299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.248168    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-e2e--71cd8499-dockercfg-kmjj5\""
Nov 02 08:06:45.248299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.248195    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"routing-overlay\""
Nov 02 08:06:45.318902 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.318862    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qdbs\" (UniqueName: \"kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-kube-api-access-9qdbs\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.319127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.319110    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.319171 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.319147    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.319277 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.319261    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.420582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.420542    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.420582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.420594    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.420818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.420684    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.420818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.420769    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qdbs\" (UniqueName: \"kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-kube-api-access-9qdbs\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.421492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.421465    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.421645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.421538    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.423202 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.423179    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.444660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.444626    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qdbs\" (UniqueName: \"kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-kube-api-access-9qdbs\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.538419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.538330    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:45.858633 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:06:45.858593    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a930b7_1dfd_4c88_a6b3_6767d7c7e8d9.slice/crio-8e750c08775fca541806c2997c50fd8904f5c4d4b69f1e0aa13f0af9025b382f WatchSource:0}: Error finding container 8e750c08775fca541806c2997c50fd8904f5c4d4b69f1e0aa13f0af9025b382f: Status 404 returned error can't find the container with id 8e750c08775fca541806c2997c50fd8904f5c4d4b69f1e0aa13f0af9025b382f
Nov 02 08:06:45.860458 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:45.860395    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"]
Nov 02 08:06:46.045247 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:46.045195    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f" event={"ID":"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9","Type":"ContainerStarted","Data":"e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785"}
Nov 02 08:06:46.045734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:46.045275    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f" event={"ID":"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9","Type":"ContainerStarted","Data":"8e750c08775fca541806c2997c50fd8904f5c4d4b69f1e0aa13f0af9025b382f"}
Nov 02 08:06:46.045734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:46.045304    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:46.087820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:46.087764    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f" podStartSLOduration=1.087749017 podStartE2EDuration="1.087749017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:46.086843775 +0000 UTC m=+2556.777170745" watchObservedRunningTime="2026-11-02 08:06:46.087749017 +0000 UTC m=+2556.778075981"
Nov 02 08:06:51.427087 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:51.426990    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"]
Nov 02 08:06:51.430939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:51.430566    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:51.441808 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:51.441755    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"]
Nov 02 08:06:55.817937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.817900    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"]
Nov 02 08:06:55.822386 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.822362    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:55.839050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.839025    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"e2e-cmp-legacy-3ab9a5-provider-cred\""
Nov 02 08:06:55.852111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.852077    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"kube-root-ca.crt\""
Nov 02 08:06:55.852306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.852092    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"]
Nov 02 08:06:55.852306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.852163    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"routing-overlay\""
Nov 02 08:06:55.852445 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.852316    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"openshift-service-ca.crt\""
Nov 02 08:06:55.871252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.858735    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0f2e06\""
Nov 02 08:06:55.936255 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.936221    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:55.936255 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.936265    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:55.936498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.936386    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ndc9\" (UniqueName: \"kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-kube-api-access-4ndc9\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:55.936543 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.936511    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:55.959982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:55.950353    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"payload-processing-external-model-e2e--9b11fa18-dockercfg-qzj28\""
Nov 02 08:06:56.041913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.040865    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:56.041913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.041169    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:56.041913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.041199    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:56.041913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.041269    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ndc9\" (UniqueName: \"kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-kube-api-access-4ndc9\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:56.042642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.042614    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:56.043432 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.043401    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:56.087998 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.068518    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:56.116263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.116222    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ndc9\" (UniqueName: \"kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-kube-api-access-4ndc9\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:56.140265 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.140226    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:56.427961 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.427918    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw" podUID="d5b48a31-4253-4e82-96ff-38685f281f68" containerName="payload-processing" containerID="cri-o://88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f" gracePeriod=25
Nov 02 08:06:56.442930 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.442877    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m" podUID="62066a90-e562-4670-b48f-0b0ae1c5de7e" containerName="payload-processing" containerID="cri-o://34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601" gracePeriod=25
Nov 02 08:06:56.467127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.467097    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"]
Nov 02 08:06:56.481862 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:06:56.481832    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486dad79_0ce8_42ce_83fa_746ab049600d.slice/crio-f001e2a581494ef04e43aee48363694f3868f2e7868d32bb36a9390e12fb11cc WatchSource:0}: Error finding container f001e2a581494ef04e43aee48363694f3868f2e7868d32bb36a9390e12fb11cc: Status 404 returned error can't find the container with id f001e2a581494ef04e43aee48363694f3868f2e7868d32bb36a9390e12fb11cc
Nov 02 08:06:56.743802 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.743770    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:56.798930 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.798880    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:06:56.860878 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.860830    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-provider-credentials\") pod \"d5b48a31-4253-4e82-96ff-38685f281f68\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") "
Nov 02 08:06:56.861424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.861018    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-kube-api-access-4xvdx\" (UniqueName: \"kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-kube-api-access-4xvdx\") pod \"62066a90-e562-4670-b48f-0b0ae1c5de7e\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") "
Nov 02 08:06:56.861424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.861109    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-provider-credentials\") pod \"62066a90-e562-4670-b48f-0b0ae1c5de7e\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") "
Nov 02 08:06:56.861424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.861180    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-routing-overlay-volume\") pod \"62066a90-e562-4670-b48f-0b0ae1c5de7e\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") "
Nov 02 08:06:56.861424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.861290    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-plugins-config-volume\") pod \"62066a90-e562-4670-b48f-0b0ae1c5de7e\" (UID: \"62066a90-e562-4670-b48f-0b0ae1c5de7e\") "
Nov 02 08:06:56.861424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.861334    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-kube-api-access-fhqhn\" (UniqueName: \"kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-kube-api-access-fhqhn\") pod \"d5b48a31-4253-4e82-96ff-38685f281f68\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") "
Nov 02 08:06:56.861424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.861365    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-routing-overlay-volume\") pod \"d5b48a31-4253-4e82-96ff-38685f281f68\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") "
Nov 02 08:06:56.861424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.861408    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-plugins-config-volume\") pod \"d5b48a31-4253-4e82-96ff-38685f281f68\" (UID: \"d5b48a31-4253-4e82-96ff-38685f281f68\") "
Nov 02 08:06:56.862046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.862020    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-routing-overlay-volume" pod "d5b48a31-4253-4e82-96ff-38685f281f68" (UID: "d5b48a31-4253-4e82-96ff-38685f281f68"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:56.862167 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.862053    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-routing-overlay-volume" pod "62066a90-e562-4670-b48f-0b0ae1c5de7e" (UID: "62066a90-e562-4670-b48f-0b0ae1c5de7e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:56.862531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.862494    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-plugins-config-volume" pod "62066a90-e562-4670-b48f-0b0ae1c5de7e" (UID: "62066a90-e562-4670-b48f-0b0ae1c5de7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:56.862628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.862574    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-plugins-config-volume" pod "d5b48a31-4253-4e82-96ff-38685f281f68" (UID: "d5b48a31-4253-4e82-96ff-38685f281f68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:06:56.865016 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.864925    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-kube-api-access-4xvdx" pod "62066a90-e562-4670-b48f-0b0ae1c5de7e" (UID: "62066a90-e562-4670-b48f-0b0ae1c5de7e"). InnerVolumeSpecName "kube-api-access-4xvdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:56.866323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.866265    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-provider-credentials" pod "d5b48a31-4253-4e82-96ff-38685f281f68" (UID: "d5b48a31-4253-4e82-96ff-38685f281f68"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:56.866323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.866301    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-kube-api-access-fhqhn" pod "d5b48a31-4253-4e82-96ff-38685f281f68" (UID: "d5b48a31-4253-4e82-96ff-38685f281f68"). InnerVolumeSpecName "kube-api-access-fhqhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:56.867544 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.867516    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-provider-credentials" pod "62066a90-e562-4670-b48f-0b0ae1c5de7e" (UID: "62066a90-e562-4670-b48f-0b0ae1c5de7e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:06:56.963766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.963662    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:56.963766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.963706    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62066a90-e562-4670-b48f-0b0ae1c5de7e-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:56.963766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.963720    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhqhn\" (UniqueName: \"kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-kube-api-access-fhqhn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:56.963766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.963734    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:56.963766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.963746    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b48a31-4253-4e82-96ff-38685f281f68-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:56.963766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.963757    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5b48a31-4253-4e82-96ff-38685f281f68-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:56.964141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.963772    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xvdx\" (UniqueName: \"kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-kube-api-access-4xvdx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:56.964141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:56.963784    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62066a90-e562-4670-b48f-0b0ae1c5de7e-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:06:57.057731 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.057695    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:06:57.195529 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.195489    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45" event={"ID":"486dad79-0ce8-42ce-83fa-746ab049600d","Type":"ContainerStarted","Data":"b93cf9151c91d8304a7d5c0d225adb152780633d4c1e6e2cfd767f3576f18ae4"}
Nov 02 08:06:57.195716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.195550    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45" event={"ID":"486dad79-0ce8-42ce-83fa-746ab049600d","Type":"ContainerStarted","Data":"f001e2a581494ef04e43aee48363694f3868f2e7868d32bb36a9390e12fb11cc"}
Nov 02 08:06:57.195716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.195593    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:06:57.197391 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.197354    4898 generic.go:356] "Generic (PLEG): container finished" podID="d5b48a31-4253-4e82-96ff-38685f281f68" containerID="88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f" exitCode=0
Nov 02 08:06:57.197391 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.197392    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"
Nov 02 08:06:57.197647 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.197446    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw" event={"ID":"d5b48a31-4253-4e82-96ff-38685f281f68","Type":"ContainerDied","Data":"88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f"}
Nov 02 08:06:57.197839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.197809    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw" event={"ID":"d5b48a31-4253-4e82-96ff-38685f281f68","Type":"ContainerDied","Data":"cc312547c025f68f54d86cbb6f90e8743c51f6f89d791a22b1af0fe81e76248e"}
Nov 02 08:06:57.197839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.197840    4898 scope.go:122] "RemoveContainer" containerID="88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f"
Nov 02 08:06:57.199974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.199717    4898 generic.go:356] "Generic (PLEG): container finished" podID="62066a90-e562-4670-b48f-0b0ae1c5de7e" containerID="34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601" exitCode=0
Nov 02 08:06:57.199974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.199778    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m" event={"ID":"62066a90-e562-4670-b48f-0b0ae1c5de7e","Type":"ContainerDied","Data":"34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601"}
Nov 02 08:06:57.199974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.199813    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m" event={"ID":"62066a90-e562-4670-b48f-0b0ae1c5de7e","Type":"ContainerDied","Data":"fc04c3f69e2a2070bc0910d997375771f9c6b230faa495cb62332d0b8b8eaec1"}
Nov 02 08:06:57.199974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.199880    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"
Nov 02 08:06:57.211246 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.211226    4898 scope.go:122] "RemoveContainer" containerID="88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f"
Nov 02 08:06:57.211551 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:06:57.211530    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f\": container with ID starting with 88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f not found: ID does not exist" containerID="88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f"
Nov 02 08:06:57.211644 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.211561    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f"} err="failed to get container status \"88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f\": rpc error: code = NotFound desc = could not find container \"88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f\": container with ID starting with 88838faa1b9bc3c3656fb9645201ef721f208e2a935532a7227fb750d55abb4f not found: ID does not exist"
Nov 02 08:06:57.211644 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.211583    4898 scope.go:122] "RemoveContainer" containerID="34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601"
Nov 02 08:06:57.222061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.222040    4898 scope.go:122] "RemoveContainer" containerID="34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601"
Nov 02 08:06:57.222336 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:06:57.222317    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601\": container with ID starting with 34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601 not found: ID does not exist" containerID="34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601"
Nov 02 08:06:57.222398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.222349    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601"} err="failed to get container status \"34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601\": rpc error: code = NotFound desc = could not find container \"34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601\": container with ID starting with 34128aee620968c63685a0cd595eaaec16b8c19829aa739e1b2494b308750601 not found: ID does not exist"
Nov 02 08:06:57.240290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.240238    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45" podStartSLOduration=2.24022444 podStartE2EDuration="2.24022444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:57.237944785 +0000 UTC m=+2567.928271749" watchObservedRunningTime="2026-11-02 08:06:57.24022444 +0000 UTC m=+2567.930551450"
Nov 02 08:06:57.277987 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.277950    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"]
Nov 02 08:06:57.290926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.290794    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-6695pxr4m"]
Nov 02 08:06:57.319693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.319651    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"]
Nov 02 08:06:57.324137 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.324108    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-656ddsfdw"]
Nov 02 08:06:57.468404 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.468366    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62066a90-e562-4670-b48f-0b0ae1c5de7e" path="/var/lib/kubelet/pods/62066a90-e562-4670-b48f-0b0ae1c5de7e/volumes"
Nov 02 08:06:57.468863 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:57.468847    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b48a31-4253-4e82-96ff-38685f281f68" path="/var/lib/kubelet/pods/d5b48a31-4253-4e82-96ff-38685f281f68/volumes"
Nov 02 08:06:58.166128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.166093    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"]
Nov 02 08:06:58.167020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.166996    4898 cpu_manager.go:414] "containerMap: removing container" podUID="62066a90-e562-4670-b48f-0b0ae1c5de7e" containerName="payload-processing"
Nov 02 08:06:58.167020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.167017    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62066a90-e562-4670-b48f-0b0ae1c5de7e" containerName="payload-processing"
Nov 02 08:06:58.167233 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.167056    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d5b48a31-4253-4e82-96ff-38685f281f68" containerName="payload-processing"
Nov 02 08:06:58.167233 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.167065    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b48a31-4253-4e82-96ff-38685f281f68" containerName="payload-processing"
Nov 02 08:06:58.167233 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.167213    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="62066a90-e562-4670-b48f-0b0ae1c5de7e" containerName="payload-processing"
Nov 02 08:06:58.167233 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.167230    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b48a31-4253-4e82-96ff-38685f281f68" containerName="payload-processing"
Nov 02 08:06:58.173073 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.173054    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:06:58.197053 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.197021    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"]
Nov 02 08:06:58.203328 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.203288    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:06:58.203870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.203611    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:06:58.204049 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.203715    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:06:58.204136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.203801    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:06:58.284304 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.284258    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c4f0760-j4flw\" (UID: \"14d90515-1225-40ec-ae96-4440767ceeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:06:58.284593 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.284574    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c4f0760-j4flw\" (UID: \"14d90515-1225-40ec-ae96-4440767ceeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:06:58.284732 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.284716    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlz2v\" (UniqueName: \"kubernetes.io/projected/14d90515-1225-40ec-ae96-4440767ceeb7-kube-api-access-nlz2v\") pod \"maas-api-revoke-sub-9c4f0760-j4flw\" (UID: \"14d90515-1225-40ec-ae96-4440767ceeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:06:58.385822 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.385778    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c4f0760-j4flw\" (UID: \"14d90515-1225-40ec-ae96-4440767ceeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:06:58.386051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.385904    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlz2v\" (UniqueName: \"kubernetes.io/projected/14d90515-1225-40ec-ae96-4440767ceeb7-kube-api-access-nlz2v\") pod \"maas-api-revoke-sub-9c4f0760-j4flw\" (UID: \"14d90515-1225-40ec-ae96-4440767ceeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:06:58.386051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.385987    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c4f0760-j4flw\" (UID: \"14d90515-1225-40ec-ae96-4440767ceeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:06:58.386524 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.386492    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c4f0760-j4flw\" (UID: \"14d90515-1225-40ec-ae96-4440767ceeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:06:58.388694 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.388665    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c4f0760-j4flw\" (UID: \"14d90515-1225-40ec-ae96-4440767ceeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:06:58.441392 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.441315    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlz2v\" (UniqueName: \"kubernetes.io/projected/14d90515-1225-40ec-ae96-4440767ceeb7-kube-api-access-nlz2v\") pod \"maas-api-revoke-sub-9c4f0760-j4flw\" (UID: \"14d90515-1225-40ec-ae96-4440767ceeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:06:58.493642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.493603    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:06:58.811814 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:06:58.811783    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d90515_1225_40ec_ae96_4440767ceeb7.slice/crio-e2218aae878d7d6e07970cc75e7c3a5ac2e4d33e0bb7eacc02cb7792414d177a WatchSource:0}: Error finding container e2218aae878d7d6e07970cc75e7c3a5ac2e4d33e0bb7eacc02cb7792414d177a: Status 404 returned error can't find the container with id e2218aae878d7d6e07970cc75e7c3a5ac2e4d33e0bb7eacc02cb7792414d177a
Nov 02 08:06:58.812024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:58.811939    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"]
Nov 02 08:06:59.223879 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:59.223830    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw" event={"ID":"14d90515-1225-40ec-ae96-4440767ceeb7","Type":"ContainerStarted","Data":"39a65db83c9c721554da47c20dd290b51d55db4e3f58fa74429464e3146dfb21"}
Nov 02 08:06:59.224313 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:59.223946    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw" event={"ID":"14d90515-1225-40ec-ae96-4440767ceeb7","Type":"ContainerStarted","Data":"e2218aae878d7d6e07970cc75e7c3a5ac2e4d33e0bb7eacc02cb7792414d177a"}
Nov 02 08:06:59.272210 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:06:59.272138    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw" podStartSLOduration=1.27211995 podStartE2EDuration="1.27211995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:06:59.265835555 +0000 UTC m=+2569.956162539" watchObservedRunningTime="2026-11-02 08:06:59.27211995 +0000 UTC m=+2569.962446916"
Nov 02 08:07:00.233491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:00.233449    4898 generic.go:356] "Generic (PLEG): container finished" podID="14d90515-1225-40ec-ae96-4440767ceeb7" containerID="39a65db83c9c721554da47c20dd290b51d55db4e3f58fa74429464e3146dfb21" exitCode=0
Nov 02 08:07:00.233942 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:00.233540    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw" event={"ID":"14d90515-1225-40ec-ae96-4440767ceeb7","Type":"ContainerDied","Data":"39a65db83c9c721554da47c20dd290b51d55db4e3f58fa74429464e3146dfb21"}
Nov 02 08:07:01.421660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:01.421628    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:07:01.529709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:01.529608    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-xks-ca\") pod \"14d90515-1225-40ec-ae96-4440767ceeb7\" (UID: \"14d90515-1225-40ec-ae96-4440767ceeb7\") "
Nov 02 08:07:01.529709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:01.529665    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-openshift-ca\") pod \"14d90515-1225-40ec-ae96-4440767ceeb7\" (UID: \"14d90515-1225-40ec-ae96-4440767ceeb7\") "
Nov 02 08:07:01.529904 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:01.529866    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d90515-1225-40ec-ae96-4440767ceeb7-kube-api-access-nlz2v\" (UniqueName: \"kubernetes.io/projected/14d90515-1225-40ec-ae96-4440767ceeb7-kube-api-access-nlz2v\") pod \"14d90515-1225-40ec-ae96-4440767ceeb7\" (UID: \"14d90515-1225-40ec-ae96-4440767ceeb7\") "
Nov 02 08:07:01.530127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:01.530098    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-openshift-ca" pod "14d90515-1225-40ec-ae96-4440767ceeb7" (UID: "14d90515-1225-40ec-ae96-4440767ceeb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:01.530623 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:01.530598    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:01.532033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:01.532004    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d90515-1225-40ec-ae96-4440767ceeb7-kube-api-access-nlz2v" pod "14d90515-1225-40ec-ae96-4440767ceeb7" (UID: "14d90515-1225-40ec-ae96-4440767ceeb7"). InnerVolumeSpecName "kube-api-access-nlz2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:01.532110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:01.532031    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-xks-ca" pod "14d90515-1225-40ec-ae96-4440767ceeb7" (UID: "14d90515-1225-40ec-ae96-4440767ceeb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:07:01.632603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:01.632569    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d90515-1225-40ec-ae96-4440767ceeb7-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:01.632603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:01.632605    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlz2v\" (UniqueName: \"kubernetes.io/projected/14d90515-1225-40ec-ae96-4440767ceeb7-kube-api-access-nlz2v\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:02.253969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:02.253937    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"
Nov 02 08:07:02.254153 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:02.253928    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw" event={"ID":"14d90515-1225-40ec-ae96-4440767ceeb7","Type":"ContainerDied","Data":"e2218aae878d7d6e07970cc75e7c3a5ac2e4d33e0bb7eacc02cb7792414d177a"}
Nov 02 08:07:02.254153 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:02.254068    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2218aae878d7d6e07970cc75e7c3a5ac2e4d33e0bb7eacc02cb7792414d177a"
Nov 02 08:07:02.582127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:02.582021    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"]
Nov 02 08:07:07.582553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:07.582511    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4" podUID="e301fa34-3c7e-4b26-bef8-612a0925fb2b" containerName="payload-processing" containerID="cri-o://efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8" gracePeriod=25
Nov 02 08:07:07.887451 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:07.887419    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:07:08.035900 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.035855    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-kube-api-access-dzwvq\" (UniqueName: \"kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-kube-api-access-dzwvq\") pod \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") "
Nov 02 08:07:08.036132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.036001    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-plugins-config-volume\") pod \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") "
Nov 02 08:07:08.036132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.036096    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-routing-overlay-volume\") pod \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") "
Nov 02 08:07:08.036258 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.036158    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-provider-credentials\") pod \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\" (UID: \"e301fa34-3c7e-4b26-bef8-612a0925fb2b\") "
Nov 02 08:07:08.036493 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.036459    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-routing-overlay-volume" pod "e301fa34-3c7e-4b26-bef8-612a0925fb2b" (UID: "e301fa34-3c7e-4b26-bef8-612a0925fb2b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:08.036556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.036470    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-plugins-config-volume" pod "e301fa34-3c7e-4b26-bef8-612a0925fb2b" (UID: "e301fa34-3c7e-4b26-bef8-612a0925fb2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:08.036865 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.036840    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:08.036944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.036872    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e301fa34-3c7e-4b26-bef8-612a0925fb2b-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:08.038266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.038238    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-kube-api-access-dzwvq" pod "e301fa34-3c7e-4b26-bef8-612a0925fb2b" (UID: "e301fa34-3c7e-4b26-bef8-612a0925fb2b"). InnerVolumeSpecName "kube-api-access-dzwvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:08.038568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.038542    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-provider-credentials" pod "e301fa34-3c7e-4b26-bef8-612a0925fb2b" (UID: "e301fa34-3c7e-4b26-bef8-612a0925fb2b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:08.140055 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.139967    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwvq\" (UniqueName: \"kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-kube-api-access-dzwvq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:08.140297 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.140275    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e301fa34-3c7e-4b26-bef8-612a0925fb2b-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:08.221464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.219712    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:07:08.318785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.318725    4898 generic.go:356] "Generic (PLEG): container finished" podID="e301fa34-3c7e-4b26-bef8-612a0925fb2b" containerID="efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8" exitCode=0
Nov 02 08:07:08.319428 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.319344    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4" event={"ID":"e301fa34-3c7e-4b26-bef8-612a0925fb2b","Type":"ContainerDied","Data":"efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8"}
Nov 02 08:07:08.319680 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.319642    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4" event={"ID":"e301fa34-3c7e-4b26-bef8-612a0925fb2b","Type":"ContainerDied","Data":"7c38906e89dc977f93c364edd82646881b599e31a9a4400805f4547ab923123c"}
Nov 02 08:07:08.319852 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.319835    4898 scope.go:122] "RemoveContainer" containerID="efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8"
Nov 02 08:07:08.320252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.320232    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"
Nov 02 08:07:08.556204 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.556177    4898 scope.go:122] "RemoveContainer" containerID="efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8"
Nov 02 08:07:08.556622 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:07:08.556593    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8\": container with ID starting with efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8 not found: ID does not exist" containerID="efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8"
Nov 02 08:07:08.556741 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.556632    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8"} err="failed to get container status \"efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8\": rpc error: code = NotFound desc = could not find container \"efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8\": container with ID starting with efcf6bc850348a8f0e0291ec01d03838f92d8ba12f0316fa97271dba986144c8 not found: ID does not exist"
Nov 02 08:07:08.613237 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.613191    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"]
Nov 02 08:07:08.617508 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:08.617478    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-643355/payload-processing-external-model-e2e-extmod-b-643355-74bdswlm4"]
Nov 02 08:07:09.481709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:09.481669    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e301fa34-3c7e-4b26-bef8-612a0925fb2b" path="/var/lib/kubelet/pods/e301fa34-3c7e-4b26-bef8-612a0925fb2b/volumes"
Nov 02 08:07:10.272039 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.272000    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"]
Nov 02 08:07:10.273804 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.273774    4898 cpu_manager.go:414] "containerMap: removing container" podUID="14d90515-1225-40ec-ae96-4440767ceeb7" containerName="revoke-keys"
Nov 02 08:07:10.274010 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.273991    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d90515-1225-40ec-ae96-4440767ceeb7" containerName="revoke-keys"
Nov 02 08:07:10.274228 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.274214    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e301fa34-3c7e-4b26-bef8-612a0925fb2b" containerName="payload-processing"
Nov 02 08:07:10.274334 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.274323    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e301fa34-3c7e-4b26-bef8-612a0925fb2b" containerName="payload-processing"
Nov 02 08:07:10.274616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.274598    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d90515-1225-40ec-ae96-4440767ceeb7" containerName="revoke-keys"
Nov 02 08:07:10.274764 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.274751    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e301fa34-3c7e-4b26-bef8-612a0925fb2b" containerName="payload-processing"
Nov 02 08:07:10.307157 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.307125    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"]
Nov 02 08:07:10.307537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.307460    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.364203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.364124    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.364414 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.364227    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.364414 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.364315    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"routing-overlay\""
Nov 02 08:07:10.364414 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.364334    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.364414 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.364390    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"payload-processing-external-model-e2e--c1e7f36e-dockercfg-stf7d\""
Nov 02 08:07:10.364672 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.364425    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bw7b\" (UniqueName: \"kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-kube-api-access-7bw7b\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.364672 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.364473    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ea9556\""
Nov 02 08:07:10.364672 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.364492    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"kube-root-ca.crt\""
Nov 02 08:07:10.364672 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.364559    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"openshift-service-ca.crt\""
Nov 02 08:07:10.364672 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.364576    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"agc-cred-a8ddd0a5d184\""
Nov 02 08:07:10.466711 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.466662    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bw7b\" (UniqueName: \"kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-kube-api-access-7bw7b\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.466954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.466935    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.467011 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.466996    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.467114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.467096    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.467678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.467646    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.467935 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.467912    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.469606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.469586    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.505545 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.505505    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bw7b\" (UniqueName: \"kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-kube-api-access-7bw7b\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.674418 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.674378    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:10.712833 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.712799    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"]
Nov 02 08:07:10.717801 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.717775    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:10.745573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.744758    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:07:10.745573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.744900    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:07:10.745573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.745175    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:07:10.745573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.745228    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:07:10.745973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.745572    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"]
Nov 02 08:07:10.773548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.772945    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh6tl\" (UniqueName: \"kubernetes.io/projected/84e2015d-3712-43ef-bd55-8f42af1dac59-kube-api-access-gh6tl\") pod \"maas-api-revoke-sub-07691d98-zl9hm\" (UID: \"84e2015d-3712-43ef-bd55-8f42af1dac59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:10.773548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.773051    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07691d98-zl9hm\" (UID: \"84e2015d-3712-43ef-bd55-8f42af1dac59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:10.773548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.773284    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07691d98-zl9hm\" (UID: \"84e2015d-3712-43ef-bd55-8f42af1dac59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:10.874318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.874277    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07691d98-zl9hm\" (UID: \"84e2015d-3712-43ef-bd55-8f42af1dac59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:10.874559 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.874538    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07691d98-zl9hm\" (UID: \"84e2015d-3712-43ef-bd55-8f42af1dac59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:10.875019 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.874999    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07691d98-zl9hm\" (UID: \"84e2015d-3712-43ef-bd55-8f42af1dac59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:10.875178 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.875156    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6tl\" (UniqueName: \"kubernetes.io/projected/84e2015d-3712-43ef-bd55-8f42af1dac59-kube-api-access-gh6tl\") pod \"maas-api-revoke-sub-07691d98-zl9hm\" (UID: \"84e2015d-3712-43ef-bd55-8f42af1dac59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:10.877067 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.877042    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07691d98-zl9hm\" (UID: \"84e2015d-3712-43ef-bd55-8f42af1dac59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:10.899328 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.899292    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6tl\" (UniqueName: \"kubernetes.io/projected/84e2015d-3712-43ef-bd55-8f42af1dac59-kube-api-access-gh6tl\") pod \"maas-api-revoke-sub-07691d98-zl9hm\" (UID: \"84e2015d-3712-43ef-bd55-8f42af1dac59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:10.959806 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.959723    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"]
Nov 02 08:07:10.964422 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.964399    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:10.999264 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.999228    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"]
Nov 02 08:07:10.999522 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:10.999506    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"agc-cred-aadca540d07d\""
Nov 02 08:07:11.011697 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.011661    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"]
Nov 02 08:07:11.017255 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:07:11.017226    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced955e6_ea98_48ad_82d2_31eea58d8bd4.slice/crio-79d3095715a672fe569ab704c8398354339b565ed6dfdf5a467a67df122f84f3 WatchSource:0}: Error finding container 79d3095715a672fe569ab704c8398354339b565ed6dfdf5a467a67df122f84f3: Status 404 returned error can't find the container with id 79d3095715a672fe569ab704c8398354339b565ed6dfdf5a467a67df122f84f3
Nov 02 08:07:11.038691 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.038657    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:11.078605 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.078567    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.078783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.078752    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.078939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.078919    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r22lc\" (UniqueName: \"kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-kube-api-access-r22lc\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.079133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.079110    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.180760 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.180615    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.180760 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.180729    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.181201 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.180831    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r22lc\" (UniqueName: \"kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-kube-api-access-r22lc\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.181201 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.180987    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.182471 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.182441    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.182697 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.182673    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.184863 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.184836    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.215885 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.215808    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r22lc\" (UniqueName: \"kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-kube-api-access-r22lc\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.281035 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.280983    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:11.340238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.340199    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"]
Nov 02 08:07:11.340819 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.340465    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerName="istio-proxy" containerID="cri-o://c2e753dcb2eaa01ad783ca25bed39a9e7602026a508bfd8a949f9fd5060de936" gracePeriod=30
Nov 02 08:07:11.380739 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.377570    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"]
Nov 02 08:07:11.421333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.421289    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm" event={"ID":"84e2015d-3712-43ef-bd55-8f42af1dac59","Type":"ContainerStarted","Data":"f4d24b969f7fb014e45ca2d9ab0fdaaafab70213aae5860a73ffcf9973043890"}
Nov 02 08:07:11.431443 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.431404    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p" event={"ID":"ced955e6-ea98-48ad-82d2-31eea58d8bd4","Type":"ContainerStarted","Data":"f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2"}
Nov 02 08:07:11.431691 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.431668    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p" event={"ID":"ced955e6-ea98-48ad-82d2-31eea58d8bd4","Type":"ContainerStarted","Data":"79d3095715a672fe569ab704c8398354339b565ed6dfdf5a467a67df122f84f3"}
Nov 02 08:07:11.432031 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.432008    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:11.484015 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.483881    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p" podStartSLOduration=1.483862073 podStartE2EDuration="1.483862073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:07:11.479372202 +0000 UTC m=+2582.169699219" watchObservedRunningTime="2026-11-02 08:07:11.483862073 +0000 UTC m=+2582.174189038"
Nov 02 08:07:11.621260 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:11.621194    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"]
Nov 02 08:07:12.442361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:12.442320    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p" event={"ID":"7f28770f-1e0d-484e-b0ea-874058346bb6","Type":"ContainerStarted","Data":"89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d"}
Nov 02 08:07:12.442870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:12.442397    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:12.442870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:12.442422    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p" event={"ID":"7f28770f-1e0d-484e-b0ea-874058346bb6","Type":"ContainerStarted","Data":"c124ce0052e78bedd2ff3c7b8841d7c1054529e3abd3d68a288f97b405d2726e"}
Nov 02 08:07:12.444308 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:12.444280    4898 generic.go:356] "Generic (PLEG): container finished" podID="84e2015d-3712-43ef-bd55-8f42af1dac59" containerID="ad37446093ab5fc3560e7b22ab66675d1837f8cc6234c3b67442a74a43032ef4" exitCode=0
Nov 02 08:07:12.444450 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:12.444367    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm" event={"ID":"84e2015d-3712-43ef-bd55-8f42af1dac59","Type":"ContainerDied","Data":"ad37446093ab5fc3560e7b22ab66675d1837f8cc6234c3b67442a74a43032ef4"}
Nov 02 08:07:12.488378 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:12.488308    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p" podStartSLOduration=2.488291766 podStartE2EDuration="2.488291766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:07:12.485624556 +0000 UTC m=+2583.175951553" watchObservedRunningTime="2026-11-02 08:07:12.488291766 +0000 UTC m=+2583.178618733"
Nov 02 08:07:13.644443 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:13.644413    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:13.743126 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:13.743033    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e2015d-3712-43ef-bd55-8f42af1dac59-kube-api-access-gh6tl\" (UniqueName: \"kubernetes.io/projected/84e2015d-3712-43ef-bd55-8f42af1dac59-kube-api-access-gh6tl\") pod \"84e2015d-3712-43ef-bd55-8f42af1dac59\" (UID: \"84e2015d-3712-43ef-bd55-8f42af1dac59\") "
Nov 02 08:07:13.749447 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:13.743997    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-xks-ca\") pod \"84e2015d-3712-43ef-bd55-8f42af1dac59\" (UID: \"84e2015d-3712-43ef-bd55-8f42af1dac59\") "
Nov 02 08:07:13.784920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:13.778805    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-openshift-ca\") pod \"84e2015d-3712-43ef-bd55-8f42af1dac59\" (UID: \"84e2015d-3712-43ef-bd55-8f42af1dac59\") "
Nov 02 08:07:13.784920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:13.765142    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e2015d-3712-43ef-bd55-8f42af1dac59-kube-api-access-gh6tl" pod "84e2015d-3712-43ef-bd55-8f42af1dac59" (UID: "84e2015d-3712-43ef-bd55-8f42af1dac59"). InnerVolumeSpecName "kube-api-access-gh6tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:13.784920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:13.779514    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-openshift-ca" pod "84e2015d-3712-43ef-bd55-8f42af1dac59" (UID: "84e2015d-3712-43ef-bd55-8f42af1dac59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:13.791731 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:13.791685    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh6tl\" (UniqueName: \"kubernetes.io/projected/84e2015d-3712-43ef-bd55-8f42af1dac59-kube-api-access-gh6tl\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:13.791731 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:13.791727    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:13.792254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:13.792224    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-xks-ca" pod "84e2015d-3712-43ef-bd55-8f42af1dac59" (UID: "84e2015d-3712-43ef-bd55-8f42af1dac59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:07:13.892977 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:13.892934    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e2015d-3712-43ef-bd55-8f42af1dac59-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:14.232421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:14.232380    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:07:14.232604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:14.232448    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:07:14.465653 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:14.465625    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"
Nov 02 08:07:14.465653 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:14.465641    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm" event={"ID":"84e2015d-3712-43ef-bd55-8f42af1dac59","Type":"ContainerDied","Data":"f4d24b969f7fb014e45ca2d9ab0fdaaafab70213aae5860a73ffcf9973043890"}
Nov 02 08:07:14.465976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:14.465687    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4d24b969f7fb014e45ca2d9ab0fdaaafab70213aae5860a73ffcf9973043890"
Nov 02 08:07:14.991548 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:07:14.991503    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:07:16.486612 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:16.486577    4898 generic.go:356] "Generic (PLEG): container finished" podID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerID="c2e753dcb2eaa01ad783ca25bed39a9e7602026a508bfd8a949f9fd5060de936" exitCode=0
Nov 02 08:07:16.487124 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:16.486667    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" event={"ID":"21181cd6-c08a-4db8-b18b-8ca7b0743867","Type":"ContainerDied","Data":"c2e753dcb2eaa01ad783ca25bed39a9e7602026a508bfd8a949f9fd5060de936"}
Nov 02 08:07:17.043665 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.043627    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 08:07:17.147290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.147203    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-envoy\") pod \"21181cd6-c08a-4db8-b18b-8ca7b0743867\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") "
Nov 02 08:07:17.147464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.147293    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-podinfo\") pod \"21181cd6-c08a-4db8-b18b-8ca7b0743867\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") "
Nov 02 08:07:17.147464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.147398    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-socket\") pod \"21181cd6-c08a-4db8-b18b-8ca7b0743867\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") "
Nov 02 08:07:17.147599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.147463    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21181cd6-c08a-4db8-b18b-8ca7b0743867-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21181cd6-c08a-4db8-b18b-8ca7b0743867-istiod-ca-cert\") pod \"21181cd6-c08a-4db8-b18b-8ca7b0743867\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") "
Nov 02 08:07:17.147599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.147496    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-token\" (UniqueName: \"kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-token\") pod \"21181cd6-c08a-4db8-b18b-8ca7b0743867\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") "
Nov 02 08:07:17.147714 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.147604    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-certs\") pod \"21181cd6-c08a-4db8-b18b-8ca7b0743867\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") "
Nov 02 08:07:17.147714 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.147686    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-kube-api-access-r9p9d\" (UniqueName: \"kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-kube-api-access-r9p9d\") pod \"21181cd6-c08a-4db8-b18b-8ca7b0743867\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") "
Nov 02 08:07:17.147910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.147792    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-data\") pod \"21181cd6-c08a-4db8-b18b-8ca7b0743867\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") "
Nov 02 08:07:17.147910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.147862    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-credential-socket\") pod \"21181cd6-c08a-4db8-b18b-8ca7b0743867\" (UID: \"21181cd6-c08a-4db8-b18b-8ca7b0743867\") "
Nov 02 08:07:17.148583 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.147851    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21181cd6-c08a-4db8-b18b-8ca7b0743867-istiod-ca-cert" pod "21181cd6-c08a-4db8-b18b-8ca7b0743867" (UID: "21181cd6-c08a-4db8-b18b-8ca7b0743867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:17.148704 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.148303    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-socket" pod "21181cd6-c08a-4db8-b18b-8ca7b0743867" (UID: "21181cd6-c08a-4db8-b18b-8ca7b0743867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:07:17.148704 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.148475    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-data" pod "21181cd6-c08a-4db8-b18b-8ca7b0743867" (UID: "21181cd6-c08a-4db8-b18b-8ca7b0743867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:07:17.148948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.148821    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-credential-socket" pod "21181cd6-c08a-4db8-b18b-8ca7b0743867" (UID: "21181cd6-c08a-4db8-b18b-8ca7b0743867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:07:17.149132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.149109    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-certs" pod "21181cd6-c08a-4db8-b18b-8ca7b0743867" (UID: "21181cd6-c08a-4db8-b18b-8ca7b0743867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:07:17.150207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.150169    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-envoy" pod "21181cd6-c08a-4db8-b18b-8ca7b0743867" (UID: "21181cd6-c08a-4db8-b18b-8ca7b0743867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:07:17.150521 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.150482    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-kube-api-access-r9p9d" pod "21181cd6-c08a-4db8-b18b-8ca7b0743867" (UID: "21181cd6-c08a-4db8-b18b-8ca7b0743867"). InnerVolumeSpecName "kube-api-access-r9p9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:17.150637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.150529    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-token" pod "21181cd6-c08a-4db8-b18b-8ca7b0743867" (UID: "21181cd6-c08a-4db8-b18b-8ca7b0743867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:17.151019 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.150993    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-podinfo" pod "21181cd6-c08a-4db8-b18b-8ca7b0743867" (UID: "21181cd6-c08a-4db8-b18b-8ca7b0743867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:07:17.249549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.249512    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:17.249549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.249545    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:17.249549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.249555    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:17.249825 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.249563    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21181cd6-c08a-4db8-b18b-8ca7b0743867-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:17.249825 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.249572    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:17.249825 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.249580    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:17.249825 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.249588    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9p9d\" (UniqueName: \"kubernetes.io/projected/21181cd6-c08a-4db8-b18b-8ca7b0743867-kube-api-access-r9p9d\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:17.249825 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.249598    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:17.249825 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.249607    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21181cd6-c08a-4db8-b18b-8ca7b0743867-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:17.507785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.507397    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl" event={"ID":"21181cd6-c08a-4db8-b18b-8ca7b0743867","Type":"ContainerDied","Data":"c727c01d7396ecba722cbb1a39b867ca338f58596ae02468931d794c58f54629"}
Nov 02 08:07:17.507785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.507485    4898 scope.go:122] "RemoveContainer" containerID="c2e753dcb2eaa01ad783ca25bed39a9e7602026a508bfd8a949f9fd5060de936"
Nov 02 08:07:17.507785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.507509    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"
Nov 02 08:07:17.580621 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.580587    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"]
Nov 02 08:07:17.592509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:17.592475    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-292kl"]
Nov 02 08:07:19.469542 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:19.469504    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" path="/var/lib/kubelet/pods/21181cd6-c08a-4db8-b18b-8ca7b0743867/volumes"
Nov 02 08:07:21.420134 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.420099    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"]
Nov 02 08:07:21.420877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.420856    4898 cpu_manager.go:414] "containerMap: removing container" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerName="istio-proxy"
Nov 02 08:07:21.420877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.420877    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerName="istio-proxy"
Nov 02 08:07:21.420978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.420913    4898 cpu_manager.go:414] "containerMap: removing container" podUID="84e2015d-3712-43ef-bd55-8f42af1dac59" containerName="revoke-keys"
Nov 02 08:07:21.420978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.420919    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e2015d-3712-43ef-bd55-8f42af1dac59" containerName="revoke-keys"
Nov 02 08:07:21.421049 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.421026    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="21181cd6-c08a-4db8-b18b-8ca7b0743867" containerName="istio-proxy"
Nov 02 08:07:21.421049 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.421040    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e2015d-3712-43ef-bd55-8f42af1dac59" containerName="revoke-keys"
Nov 02 08:07:21.426329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.426309    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:21.444491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.444455    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"]
Nov 02 08:07:21.450315 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.450274    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:07:21.450528 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.450424    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:07:21.450528 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.450496    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:07:21.453640 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.453615    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:07:21.520669 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.520622    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f697g\" (UniqueName: \"kubernetes.io/projected/a13c1f64-b43a-4473-a587-7fe96fa953f3-kube-api-access-f697g\") pod \"maas-api-revoke-sub-db8bd722-msg4j\" (UID: \"a13c1f64-b43a-4473-a587-7fe96fa953f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:21.520934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.520768    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db8bd722-msg4j\" (UID: \"a13c1f64-b43a-4473-a587-7fe96fa953f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:21.521053 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.521020    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db8bd722-msg4j\" (UID: \"a13c1f64-b43a-4473-a587-7fe96fa953f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:21.622480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.622420    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db8bd722-msg4j\" (UID: \"a13c1f64-b43a-4473-a587-7fe96fa953f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:21.622699 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.622568    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f697g\" (UniqueName: \"kubernetes.io/projected/a13c1f64-b43a-4473-a587-7fe96fa953f3-kube-api-access-f697g\") pod \"maas-api-revoke-sub-db8bd722-msg4j\" (UID: \"a13c1f64-b43a-4473-a587-7fe96fa953f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:21.622699 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.622662    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db8bd722-msg4j\" (UID: \"a13c1f64-b43a-4473-a587-7fe96fa953f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:21.623208 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.623176    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db8bd722-msg4j\" (UID: \"a13c1f64-b43a-4473-a587-7fe96fa953f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:21.625252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.625225    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db8bd722-msg4j\" (UID: \"a13c1f64-b43a-4473-a587-7fe96fa953f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:21.646780 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.646749    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f697g\" (UniqueName: \"kubernetes.io/projected/a13c1f64-b43a-4473-a587-7fe96fa953f3-kube-api-access-f697g\") pod \"maas-api-revoke-sub-db8bd722-msg4j\" (UID: \"a13c1f64-b43a-4473-a587-7fe96fa953f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:21.744312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:21.744229    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:22.059714 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:07:22.059681    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13c1f64_b43a_4473_a587_7fe96fa953f3.slice/crio-7242c703bd9e516359c28c2d4120de843f3f9b92b21952847fb6fca1cd3cf00b WatchSource:0}: Error finding container 7242c703bd9e516359c28c2d4120de843f3f9b92b21952847fb6fca1cd3cf00b: Status 404 returned error can't find the container with id 7242c703bd9e516359c28c2d4120de843f3f9b92b21952847fb6fca1cd3cf00b
Nov 02 08:07:22.066562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:22.066531    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"]
Nov 02 08:07:22.447472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:22.447443    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:22.563913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:22.562917    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j" event={"ID":"a13c1f64-b43a-4473-a587-7fe96fa953f3","Type":"ContainerStarted","Data":"046fd2d33455f38350a9b5f9f14cc8741f76f9b8fe414a99ac1eed3311a9ff39"}
Nov 02 08:07:22.563913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:22.562987    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j" event={"ID":"a13c1f64-b43a-4473-a587-7fe96fa953f3","Type":"ContainerStarted","Data":"7242c703bd9e516359c28c2d4120de843f3f9b92b21952847fb6fca1cd3cf00b"}
Nov 02 08:07:22.637703 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:22.637637    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j" podStartSLOduration=1.637616656 podStartE2EDuration="1.637616656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:07:22.629879915 +0000 UTC m=+2593.320206883" watchObservedRunningTime="2026-11-02 08:07:22.637616656 +0000 UTC m=+2593.327943622"
Nov 02 08:07:23.472440 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:23.472399    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:07:23.732925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:23.717795    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"]
Nov 02 08:07:23.770613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:23.770572    4898 generic.go:356] "Generic (PLEG): container finished" podID="a13c1f64-b43a-4473-a587-7fe96fa953f3" containerID="046fd2d33455f38350a9b5f9f14cc8741f76f9b8fe414a99ac1eed3311a9ff39" exitCode=0
Nov 02 08:07:23.770820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:23.770677    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j" event={"ID":"a13c1f64-b43a-4473-a587-7fe96fa953f3","Type":"ContainerDied","Data":"046fd2d33455f38350a9b5f9f14cc8741f76f9b8fe414a99ac1eed3311a9ff39"}
Nov 02 08:07:24.980526 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:24.980492    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:25.033637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:25.033542    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13c1f64-b43a-4473-a587-7fe96fa953f3-kube-api-access-f697g\" (UniqueName: \"kubernetes.io/projected/a13c1f64-b43a-4473-a587-7fe96fa953f3-kube-api-access-f697g\") pod \"a13c1f64-b43a-4473-a587-7fe96fa953f3\" (UID: \"a13c1f64-b43a-4473-a587-7fe96fa953f3\") "
Nov 02 08:07:25.033637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:25.033606    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-openshift-ca\") pod \"a13c1f64-b43a-4473-a587-7fe96fa953f3\" (UID: \"a13c1f64-b43a-4473-a587-7fe96fa953f3\") "
Nov 02 08:07:25.033884 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:25.033670    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-xks-ca\") pod \"a13c1f64-b43a-4473-a587-7fe96fa953f3\" (UID: \"a13c1f64-b43a-4473-a587-7fe96fa953f3\") "
Nov 02 08:07:25.034370 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:25.034345    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-openshift-ca" pod "a13c1f64-b43a-4473-a587-7fe96fa953f3" (UID: "a13c1f64-b43a-4473-a587-7fe96fa953f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:25.036241 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:25.036103    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13c1f64-b43a-4473-a587-7fe96fa953f3-kube-api-access-f697g" pod "a13c1f64-b43a-4473-a587-7fe96fa953f3" (UID: "a13c1f64-b43a-4473-a587-7fe96fa953f3"). InnerVolumeSpecName "kube-api-access-f697g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:25.036513 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:25.036482    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-xks-ca" pod "a13c1f64-b43a-4473-a587-7fe96fa953f3" (UID: "a13c1f64-b43a-4473-a587-7fe96fa953f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:07:25.136038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:25.135993    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f697g\" (UniqueName: \"kubernetes.io/projected/a13c1f64-b43a-4473-a587-7fe96fa953f3-kube-api-access-f697g\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:25.136038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:25.136027    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:25.136038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:25.136038    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13c1f64-b43a-4473-a587-7fe96fa953f3-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:25.791348 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:25.790769    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"
Nov 02 08:07:25.791348 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:25.790776    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j" event={"ID":"a13c1f64-b43a-4473-a587-7fe96fa953f3","Type":"ContainerDied","Data":"7242c703bd9e516359c28c2d4120de843f3f9b92b21952847fb6fca1cd3cf00b"}
Nov 02 08:07:25.791348 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:25.790926    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7242c703bd9e516359c28c2d4120de843f3f9b92b21952847fb6fca1cd3cf00b"
Nov 02 08:07:28.718521 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:28.718469    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p" podUID="ced955e6-ea98-48ad-82d2-31eea58d8bd4" containerName="payload-processing" containerID="cri-o://f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2" gracePeriod=25
Nov 02 08:07:29.000088 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.000056    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:29.095165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.095124    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-routing-overlay-volume\") pod \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") "
Nov 02 08:07:29.095394 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.095191    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-kube-api-access-7bw7b\" (UniqueName: \"kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-kube-api-access-7bw7b\") pod \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") "
Nov 02 08:07:29.095476 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.095399    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-plugins-config-volume\") pod \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") "
Nov 02 08:07:29.095728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.095706    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-provider-credentials\") pod \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\" (UID: \"ced955e6-ea98-48ad-82d2-31eea58d8bd4\") "
Nov 02 08:07:29.095833 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.095816    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-plugins-config-volume" pod "ced955e6-ea98-48ad-82d2-31eea58d8bd4" (UID: "ced955e6-ea98-48ad-82d2-31eea58d8bd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:29.095904 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.095859    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-routing-overlay-volume" pod "ced955e6-ea98-48ad-82d2-31eea58d8bd4" (UID: "ced955e6-ea98-48ad-82d2-31eea58d8bd4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:29.096444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.096418    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:29.096444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.096446    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced955e6-ea98-48ad-82d2-31eea58d8bd4-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:29.098275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.098238    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-kube-api-access-7bw7b" pod "ced955e6-ea98-48ad-82d2-31eea58d8bd4" (UID: "ced955e6-ea98-48ad-82d2-31eea58d8bd4"). InnerVolumeSpecName "kube-api-access-7bw7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:29.098395 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.098348    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-provider-credentials" pod "ced955e6-ea98-48ad-82d2-31eea58d8bd4" (UID: "ced955e6-ea98-48ad-82d2-31eea58d8bd4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:29.197933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.197873    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:29.197933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.197924    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bw7b\" (UniqueName: \"kubernetes.io/projected/ced955e6-ea98-48ad-82d2-31eea58d8bd4-kube-api-access-7bw7b\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:29.829419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.829321    4898 generic.go:356] "Generic (PLEG): container finished" podID="ced955e6-ea98-48ad-82d2-31eea58d8bd4" containerID="f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2" exitCode=0
Nov 02 08:07:29.829419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.829378    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"
Nov 02 08:07:29.829419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.829409    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p" event={"ID":"ced955e6-ea98-48ad-82d2-31eea58d8bd4","Type":"ContainerDied","Data":"f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2"}
Nov 02 08:07:29.830147 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.829471    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p" event={"ID":"ced955e6-ea98-48ad-82d2-31eea58d8bd4","Type":"ContainerDied","Data":"79d3095715a672fe569ab704c8398354339b565ed6dfdf5a467a67df122f84f3"}
Nov 02 08:07:29.830147 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.829499    4898 scope.go:122] "RemoveContainer" containerID="f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2"
Nov 02 08:07:29.841163 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.841138    4898 scope.go:122] "RemoveContainer" containerID="f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2"
Nov 02 08:07:29.841472 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:07:29.841445    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2\": container with ID starting with f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2 not found: ID does not exist" containerID="f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2"
Nov 02 08:07:29.841536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.841481    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2"} err="failed to get container status \"f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2\": rpc error: code = NotFound desc = could not find container \"f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2\": container with ID starting with f4684a16983ebca2a26778e55b5c461499be6840b631bc2710570c1f5d7f23e2 not found: ID does not exist"
Nov 02 08:07:29.884441 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.884400    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"]
Nov 02 08:07:29.897213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:29.897163    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-5b6bsh87p"]
Nov 02 08:07:31.466301 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:31.466265    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced955e6-ea98-48ad-82d2-31eea58d8bd4" path="/var/lib/kubelet/pods/ced955e6-ea98-48ad-82d2-31eea58d8bd4/volumes"
Nov 02 08:07:34.809158 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.809121    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"]
Nov 02 08:07:34.810107 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.810082    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a13c1f64-b43a-4473-a587-7fe96fa953f3" containerName="revoke-keys"
Nov 02 08:07:34.810182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.810109    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13c1f64-b43a-4473-a587-7fe96fa953f3" containerName="revoke-keys"
Nov 02 08:07:34.810248 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.810202    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ced955e6-ea98-48ad-82d2-31eea58d8bd4" containerName="payload-processing"
Nov 02 08:07:34.810248 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.810211    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced955e6-ea98-48ad-82d2-31eea58d8bd4" containerName="payload-processing"
Nov 02 08:07:34.810396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.810382    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced955e6-ea98-48ad-82d2-31eea58d8bd4" containerName="payload-processing"
Nov 02 08:07:34.810437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.810404    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13c1f64-b43a-4473-a587-7fe96fa953f3" containerName="revoke-keys"
Nov 02 08:07:34.815617 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.815598    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:34.832779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.832747    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"]
Nov 02 08:07:34.839605 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.839577    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:07:34.839782 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.839613    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:07:34.839782 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.839632    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:07:34.842630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.842606    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:07:34.982172 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.982122    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-863682c5-7j26k\" (UID: \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:34.982355 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.982194    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzgkh\" (UniqueName: \"kubernetes.io/projected/65bab501-f9b8-4536-9ee4-b6a34a814dc1-kube-api-access-wzgkh\") pod \"maas-api-revoke-sub-863682c5-7j26k\" (UID: \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:34.982490 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:34.982473    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-863682c5-7j26k\" (UID: \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:35.084032 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:35.083938    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-863682c5-7j26k\" (UID: \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:35.084211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:35.084047    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-863682c5-7j26k\" (UID: \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:35.084211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:35.084083    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzgkh\" (UniqueName: \"kubernetes.io/projected/65bab501-f9b8-4536-9ee4-b6a34a814dc1-kube-api-access-wzgkh\") pod \"maas-api-revoke-sub-863682c5-7j26k\" (UID: \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:35.084784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:35.084747    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-863682c5-7j26k\" (UID: \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:35.086610 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:35.086579    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-863682c5-7j26k\" (UID: \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:35.108197 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:35.108165    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzgkh\" (UniqueName: \"kubernetes.io/projected/65bab501-f9b8-4536-9ee4-b6a34a814dc1-kube-api-access-wzgkh\") pod \"maas-api-revoke-sub-863682c5-7j26k\" (UID: \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:35.131329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:35.131298    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:35.442235 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:07:35.442203    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bab501_f9b8_4536_9ee4_b6a34a814dc1.slice/crio-cf8cdc5c2020125638551e7ac8efcdfeba81458837b70eb0c74532ece666fa4f WatchSource:0}: Error finding container cf8cdc5c2020125638551e7ac8efcdfeba81458837b70eb0c74532ece666fa4f: Status 404 returned error can't find the container with id cf8cdc5c2020125638551e7ac8efcdfeba81458837b70eb0c74532ece666fa4f
Nov 02 08:07:35.445548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:35.445517    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"]
Nov 02 08:07:35.893142 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:35.893101    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k" event={"ID":"65bab501-f9b8-4536-9ee4-b6a34a814dc1","Type":"ContainerStarted","Data":"782b7f0ca6bf087f7ef8f4852dc37455398297ef9f10be7ca0e828ff2a39bf97"}
Nov 02 08:07:35.893618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:35.893172    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k" event={"ID":"65bab501-f9b8-4536-9ee4-b6a34a814dc1","Type":"ContainerStarted","Data":"cf8cdc5c2020125638551e7ac8efcdfeba81458837b70eb0c74532ece666fa4f"}
Nov 02 08:07:35.936752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:35.936695    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k" podStartSLOduration=1.9366822620000002 podStartE2EDuration="1.936682262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:07:35.934042693 +0000 UTC m=+2606.624369674" watchObservedRunningTime="2026-11-02 08:07:35.936682262 +0000 UTC m=+2606.627009225"
Nov 02 08:07:36.905088 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:36.905054    4898 generic.go:356] "Generic (PLEG): container finished" podID="65bab501-f9b8-4536-9ee4-b6a34a814dc1" containerID="782b7f0ca6bf087f7ef8f4852dc37455398297ef9f10be7ca0e828ff2a39bf97" exitCode=0
Nov 02 08:07:36.905503 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:36.905139    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k" event={"ID":"65bab501-f9b8-4536-9ee4-b6a34a814dc1","Type":"ContainerDied","Data":"782b7f0ca6bf087f7ef8f4852dc37455398297ef9f10be7ca0e828ff2a39bf97"}
Nov 02 08:07:38.094719 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.094683    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:38.228186 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.228081    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-xks-ca\") pod \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\" (UID: \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\") "
Nov 02 08:07:38.228186 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.228164    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-openshift-ca\") pod \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\" (UID: \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\") "
Nov 02 08:07:38.228440 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.228354    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bab501-f9b8-4536-9ee4-b6a34a814dc1-kube-api-access-wzgkh\" (UniqueName: \"kubernetes.io/projected/65bab501-f9b8-4536-9ee4-b6a34a814dc1-kube-api-access-wzgkh\") pod \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\" (UID: \"65bab501-f9b8-4536-9ee4-b6a34a814dc1\") "
Nov 02 08:07:38.228632 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.228608    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-openshift-ca" pod "65bab501-f9b8-4536-9ee4-b6a34a814dc1" (UID: "65bab501-f9b8-4536-9ee4-b6a34a814dc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:07:38.228885 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.228866    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:38.230434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.230402    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-xks-ca" pod "65bab501-f9b8-4536-9ee4-b6a34a814dc1" (UID: "65bab501-f9b8-4536-9ee4-b6a34a814dc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:07:38.230551 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.230529    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bab501-f9b8-4536-9ee4-b6a34a814dc1-kube-api-access-wzgkh" pod "65bab501-f9b8-4536-9ee4-b6a34a814dc1" (UID: "65bab501-f9b8-4536-9ee4-b6a34a814dc1"). InnerVolumeSpecName "kube-api-access-wzgkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:07:38.330529 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.330482    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzgkh\" (UniqueName: \"kubernetes.io/projected/65bab501-f9b8-4536-9ee4-b6a34a814dc1-kube-api-access-wzgkh\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:38.330529 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.330529    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65bab501-f9b8-4536-9ee4-b6a34a814dc1-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:07:38.923745 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.923712    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"
Nov 02 08:07:38.923974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.923721    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k" event={"ID":"65bab501-f9b8-4536-9ee4-b6a34a814dc1","Type":"ContainerDied","Data":"cf8cdc5c2020125638551e7ac8efcdfeba81458837b70eb0c74532ece666fa4f"}
Nov 02 08:07:38.923974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:07:38.923842    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf8cdc5c2020125638551e7ac8efcdfeba81458837b70eb0c74532ece666fa4f"
Nov 02 08:07:44.991798 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:07:44.991756    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:08:04.194018 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.193976    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"]
Nov 02 08:08:04.195122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.194903    4898 cpu_manager.go:414] "containerMap: removing container" podUID="65bab501-f9b8-4536-9ee4-b6a34a814dc1" containerName="revoke-keys"
Nov 02 08:08:04.195122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.194921    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bab501-f9b8-4536-9ee4-b6a34a814dc1" containerName="revoke-keys"
Nov 02 08:08:04.195122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.195054    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bab501-f9b8-4536-9ee4-b6a34a814dc1" containerName="revoke-keys"
Nov 02 08:08:04.200336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.200316    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.217492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.217456    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"kube-root-ca.crt\""
Nov 02 08:08:04.220059 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.220030    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"]
Nov 02 08:08:04.225786 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.225741    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-696c7a\""
Nov 02 08:08:04.225786 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.225751    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"routing-overlay\""
Nov 02 08:08:04.227002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.226979    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"agc-cred-3d073bae94b5\""
Nov 02 08:08:04.227214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.227196    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"openshift-service-ca.crt\""
Nov 02 08:08:04.229959 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.229934    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696c7a\"/\"payload-processing-external-model-e2e--4bba7e07-dockercfg-xjx8h\""
Nov 02 08:08:04.244829 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.244795    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.245038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.244863    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.245038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.244951    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.245038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.245019    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs4k2\" (UniqueName: \"kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-kube-api-access-hs4k2\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.382911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.361760    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.382911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.361856    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.382911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.361936    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.382911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.361996    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4k2\" (UniqueName: \"kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-kube-api-access-hs4k2\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.382911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.363623    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.382911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.364302    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.400941 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.400868    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.404348 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.404318    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs4k2\" (UniqueName: \"kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-kube-api-access-hs4k2\") pod \"payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") " pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.520807 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.520708    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:04.863448 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:08:04.863419    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27383256_374d_49e2_bdf3_74a20ce00532.slice/crio-b0880c08902bc4fc408bc3a5fa1f4230a7f4aea4671695eded03574755c02820 WatchSource:0}: Error finding container b0880c08902bc4fc408bc3a5fa1f4230a7f4aea4671695eded03574755c02820: Status 404 returned error can't find the container with id b0880c08902bc4fc408bc3a5fa1f4230a7f4aea4671695eded03574755c02820
Nov 02 08:08:04.864360 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:04.864324    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"]
Nov 02 08:08:05.224411 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:05.224373    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6" event={"ID":"27383256-374d-49e2-bdf3-74a20ce00532","Type":"ContainerStarted","Data":"1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97"}
Nov 02 08:08:05.225042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:05.224474    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6" event={"ID":"27383256-374d-49e2-bdf3-74a20ce00532","Type":"ContainerStarted","Data":"b0880c08902bc4fc408bc3a5fa1f4230a7f4aea4671695eded03574755c02820"}
Nov 02 08:08:05.225042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:05.224840    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:05.287590 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:05.287516    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6" podStartSLOduration=1.287494467 podStartE2EDuration="1.287494467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:08:05.279790362 +0000 UTC m=+2635.970117350" watchObservedRunningTime="2026-11-02 08:08:05.287494467 +0000 UTC m=+2635.977821432"
Nov 02 08:08:06.023187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.023149    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"]
Nov 02 08:08:06.028026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.028006    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:06.047911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.045315    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:08:06.047911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.045953    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:08:06.057909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.056425    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:08:06.057909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.056538    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:08:06.066026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.065144    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"]
Nov 02 08:08:06.087673 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.087636    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69769763-f064-474f-9ddf-300b7f499a53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580227db-g2krw\" (UID: \"69769763-f064-474f-9ddf-300b7f499a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:06.087877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.087768    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk82v\" (UniqueName: \"kubernetes.io/projected/69769763-f064-474f-9ddf-300b7f499a53-kube-api-access-pk82v\") pod \"maas-api-revoke-sub-580227db-g2krw\" (UID: \"69769763-f064-474f-9ddf-300b7f499a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:06.088090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.088064    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69769763-f064-474f-9ddf-300b7f499a53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580227db-g2krw\" (UID: \"69769763-f064-474f-9ddf-300b7f499a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:06.189084 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.189049    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69769763-f064-474f-9ddf-300b7f499a53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580227db-g2krw\" (UID: \"69769763-f064-474f-9ddf-300b7f499a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:06.189296 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.189107    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69769763-f064-474f-9ddf-300b7f499a53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580227db-g2krw\" (UID: \"69769763-f064-474f-9ddf-300b7f499a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:06.189296 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.189189    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk82v\" (UniqueName: \"kubernetes.io/projected/69769763-f064-474f-9ddf-300b7f499a53-kube-api-access-pk82v\") pod \"maas-api-revoke-sub-580227db-g2krw\" (UID: \"69769763-f064-474f-9ddf-300b7f499a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:06.189810 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.189783    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69769763-f064-474f-9ddf-300b7f499a53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-580227db-g2krw\" (UID: \"69769763-f064-474f-9ddf-300b7f499a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:06.191954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.191930    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69769763-f064-474f-9ddf-300b7f499a53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-580227db-g2krw\" (UID: \"69769763-f064-474f-9ddf-300b7f499a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:06.212687 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.212656    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk82v\" (UniqueName: \"kubernetes.io/projected/69769763-f064-474f-9ddf-300b7f499a53-kube-api-access-pk82v\") pod \"maas-api-revoke-sub-580227db-g2krw\" (UID: \"69769763-f064-474f-9ddf-300b7f499a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:06.344802 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.344716    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:06.659200 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:08:06.659163    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69769763_f064_474f_9ddf_300b7f499a53.slice/crio-eb402131a3027c955e852aa2e22bdf2137d07d0ca2c8c015689140d5d61c76ea WatchSource:0}: Error finding container eb402131a3027c955e852aa2e22bdf2137d07d0ca2c8c015689140d5d61c76ea: Status 404 returned error can't find the container with id eb402131a3027c955e852aa2e22bdf2137d07d0ca2c8c015689140d5d61c76ea
Nov 02 08:08:06.666964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:06.666932    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"]
Nov 02 08:08:07.246380 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.246341    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw" event={"ID":"69769763-f064-474f-9ddf-300b7f499a53","Type":"ContainerStarted","Data":"eb402131a3027c955e852aa2e22bdf2137d07d0ca2c8c015689140d5d61c76ea"}
Nov 02 08:08:07.771265 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.771234    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"]
Nov 02 08:08:07.776239 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.776215    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:07.796452 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.796418    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"]
Nov 02 08:08:07.802970 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.802921    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"agc-cred-1635ca0f4f34\""
Nov 02 08:08:07.820333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.820299    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:07.820515 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.820347    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqtn9\" (UniqueName: \"kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-kube-api-access-lqtn9\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:07.820600 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.820506    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:07.820695 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.820677    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:07.922213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.922174    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:07.922420 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.922231    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqtn9\" (UniqueName: \"kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-kube-api-access-lqtn9\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:07.922420 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.922301    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:07.922555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.922454    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:07.923090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.923065    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:07.923189 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.923119    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:07.924943 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.924919    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:07.946576 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:07.946551    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqtn9\" (UniqueName: \"kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-kube-api-access-lqtn9\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:08.142934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:08.142824    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:08.279687 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:08.260772    4898 generic.go:356] "Generic (PLEG): container finished" podID="69769763-f064-474f-9ddf-300b7f499a53" containerID="ff11e009f60dfed34a84f15d60fcdc2bfc55c243ba8e2a99b3c1e0d0188e22cc" exitCode=0
Nov 02 08:08:08.279687 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:08.260863    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw" event={"ID":"69769763-f064-474f-9ddf-300b7f499a53","Type":"ContainerDied","Data":"ff11e009f60dfed34a84f15d60fcdc2bfc55c243ba8e2a99b3c1e0d0188e22cc"}
Nov 02 08:08:08.475632 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:08.475602    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"]
Nov 02 08:08:09.270852 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.270807    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2" event={"ID":"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce","Type":"ContainerStarted","Data":"3d0c3e79097d5541cd515946f46d9d9986c22d60f9672126f7d9d351810aefab"}
Nov 02 08:08:09.271433 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.270873    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2" event={"ID":"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce","Type":"ContainerStarted","Data":"3d0668f244ba8a4c09442ff6c233acc87b030a5ad9855a9931df7cbaa83817d5"}
Nov 02 08:08:09.314297 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.314242    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2" podStartSLOduration=2.314225659 podStartE2EDuration="2.314225659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:08:09.311524606 +0000 UTC m=+2640.001851570" watchObservedRunningTime="2026-11-02 08:08:09.314225659 +0000 UTC m=+2640.004552622"
Nov 02 08:08:09.444103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.444073    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:09.546794 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.546393    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69769763-f064-474f-9ddf-300b7f499a53-kube-api-access-pk82v\" (UniqueName: \"kubernetes.io/projected/69769763-f064-474f-9ddf-300b7f499a53-kube-api-access-pk82v\") pod \"69769763-f064-474f-9ddf-300b7f499a53\" (UID: \"69769763-f064-474f-9ddf-300b7f499a53\") "
Nov 02 08:08:09.546794 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.546474    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69769763-f064-474f-9ddf-300b7f499a53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69769763-f064-474f-9ddf-300b7f499a53-maas-api-xks-ca\") pod \"69769763-f064-474f-9ddf-300b7f499a53\" (UID: \"69769763-f064-474f-9ddf-300b7f499a53\") "
Nov 02 08:08:09.547108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.546826    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69769763-f064-474f-9ddf-300b7f499a53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69769763-f064-474f-9ddf-300b7f499a53-maas-api-openshift-ca\") pod \"69769763-f064-474f-9ddf-300b7f499a53\" (UID: \"69769763-f064-474f-9ddf-300b7f499a53\") "
Nov 02 08:08:09.547803 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.547772    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69769763-f064-474f-9ddf-300b7f499a53-maas-api-openshift-ca" pod "69769763-f064-474f-9ddf-300b7f499a53" (UID: "69769763-f064-474f-9ddf-300b7f499a53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:08:09.549465 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.549434    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69769763-f064-474f-9ddf-300b7f499a53-kube-api-access-pk82v" pod "69769763-f064-474f-9ddf-300b7f499a53" (UID: "69769763-f064-474f-9ddf-300b7f499a53"). InnerVolumeSpecName "kube-api-access-pk82v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:09.549784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.549754    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69769763-f064-474f-9ddf-300b7f499a53-maas-api-xks-ca" pod "69769763-f064-474f-9ddf-300b7f499a53" (UID: "69769763-f064-474f-9ddf-300b7f499a53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:08:09.648504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.648470    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk82v\" (UniqueName: \"kubernetes.io/projected/69769763-f064-474f-9ddf-300b7f499a53-kube-api-access-pk82v\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:09.648504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.648501    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69769763-f064-474f-9ddf-300b7f499a53-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:09.648504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:09.648510    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69769763-f064-474f-9ddf-300b7f499a53-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:10.283657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:10.283610    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw" event={"ID":"69769763-f064-474f-9ddf-300b7f499a53","Type":"ContainerDied","Data":"eb402131a3027c955e852aa2e22bdf2137d07d0ca2c8c015689140d5d61c76ea"}
Nov 02 08:08:10.283657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:10.283653    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"
Nov 02 08:08:10.284187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:10.283683    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb402131a3027c955e852aa2e22bdf2137d07d0ca2c8c015689140d5d61c76ea"
Nov 02 08:08:10.284187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:10.283868    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:14.990458 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:08:14.990416    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:08:16.240011 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.239971    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:08:16.300276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.300243    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"]
Nov 02 08:08:16.301045 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.301029    4898 cpu_manager.go:414] "containerMap: removing container" podUID="69769763-f064-474f-9ddf-300b7f499a53" containerName="revoke-keys"
Nov 02 08:08:16.301103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.301046    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69769763-f064-474f-9ddf-300b7f499a53" containerName="revoke-keys"
Nov 02 08:08:16.301195 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.301186    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="69769763-f064-474f-9ddf-300b7f499a53" containerName="revoke-keys"
Nov 02 08:08:16.306556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.306529    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:16.328772 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.328742    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:08:16.328987 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.328874    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:08:16.329675 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.329651    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"]
Nov 02 08:08:16.332012 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.331981    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:08:16.332188 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.332083    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:08:16.366744 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.366702    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fa503cd-zxhwx\" (UID: \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:16.366968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.366824    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdd7v\" (UniqueName: \"kubernetes.io/projected/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-kube-api-access-vdd7v\") pod \"maas-api-revoke-sub-9fa503cd-zxhwx\" (UID: \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:16.366968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.366948    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fa503cd-zxhwx\" (UID: \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:16.468654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.468615    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdd7v\" (UniqueName: \"kubernetes.io/projected/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-kube-api-access-vdd7v\") pod \"maas-api-revoke-sub-9fa503cd-zxhwx\" (UID: \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:16.468847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.468685    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fa503cd-zxhwx\" (UID: \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:16.468969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.468847    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fa503cd-zxhwx\" (UID: \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:16.469624 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.469599    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fa503cd-zxhwx\" (UID: \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:16.471407 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.471386    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fa503cd-zxhwx\" (UID: \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:16.492863 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.492788    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdd7v\" (UniqueName: \"kubernetes.io/projected/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-kube-api-access-vdd7v\") pod \"maas-api-revoke-sub-9fa503cd-zxhwx\" (UID: \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:16.631870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.631833    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:16.961445 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:16.961412    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"]
Nov 02 08:08:16.961800 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:08:16.961777    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984f0c6e_8b5d_4599_bc5a_8414b26d21c1.slice/crio-47af61dca28189e1712586d27242bc2436e347f0c0c3418003ee36f8be1b41c1 WatchSource:0}: Error finding container 47af61dca28189e1712586d27242bc2436e347f0c0c3418003ee36f8be1b41c1: Status 404 returned error can't find the container with id 47af61dca28189e1712586d27242bc2436e347f0c0c3418003ee36f8be1b41c1
Nov 02 08:08:17.291589 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:17.291428    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"]
Nov 02 08:08:17.357602 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:17.357535    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx" event={"ID":"984f0c6e-8b5d-4599-bc5a-8414b26d21c1","Type":"ContainerStarted","Data":"47af61dca28189e1712586d27242bc2436e347f0c0c3418003ee36f8be1b41c1"}
Nov 02 08:08:18.367839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:18.367750    4898 generic.go:356] "Generic (PLEG): container finished" podID="984f0c6e-8b5d-4599-bc5a-8414b26d21c1" containerID="f7d48ae6e93dd893a069deb2816b4fa32a388aff0345cb11b59c5433ba57966e" exitCode=0
Nov 02 08:08:18.368336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:18.367836    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx" event={"ID":"984f0c6e-8b5d-4599-bc5a-8414b26d21c1","Type":"ContainerDied","Data":"f7d48ae6e93dd893a069deb2816b4fa32a388aff0345cb11b59c5433ba57966e"}
Nov 02 08:08:19.570588 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:19.570558    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:19.675915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:19.674155    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-kube-api-access-vdd7v\" (UniqueName: \"kubernetes.io/projected/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-kube-api-access-vdd7v\") pod \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\" (UID: \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\") "
Nov 02 08:08:19.675915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:19.674237    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-xks-ca\") pod \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\" (UID: \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\") "
Nov 02 08:08:19.675915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:19.674282    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-openshift-ca\") pod \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\" (UID: \"984f0c6e-8b5d-4599-bc5a-8414b26d21c1\") "
Nov 02 08:08:19.708985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:19.702028    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-openshift-ca" pod "984f0c6e-8b5d-4599-bc5a-8414b26d21c1" (UID: "984f0c6e-8b5d-4599-bc5a-8414b26d21c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:08:19.758982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:19.758942    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-xks-ca" pod "984f0c6e-8b5d-4599-bc5a-8414b26d21c1" (UID: "984f0c6e-8b5d-4599-bc5a-8414b26d21c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:08:19.760247 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:19.760199    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-kube-api-access-vdd7v" pod "984f0c6e-8b5d-4599-bc5a-8414b26d21c1" (UID: "984f0c6e-8b5d-4599-bc5a-8414b26d21c1"). InnerVolumeSpecName "kube-api-access-vdd7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:19.776972 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:19.776871    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdd7v\" (UniqueName: \"kubernetes.io/projected/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-kube-api-access-vdd7v\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:19.776972 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:19.776929    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:19.776972 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:19.776945    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984f0c6e-8b5d-4599-bc5a-8414b26d21c1-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:20.392406 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:20.392375    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"
Nov 02 08:08:20.392633 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:20.392404    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx" event={"ID":"984f0c6e-8b5d-4599-bc5a-8414b26d21c1","Type":"ContainerDied","Data":"47af61dca28189e1712586d27242bc2436e347f0c0c3418003ee36f8be1b41c1"}
Nov 02 08:08:20.392633 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:20.392455    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47af61dca28189e1712586d27242bc2436e347f0c0c3418003ee36f8be1b41c1"
Nov 02 08:08:21.298001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:21.297969    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:08:21.447510 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:21.447473    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"]
Nov 02 08:08:22.292302 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.292213    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45" podUID="486dad79-0ce8-42ce-83fa-746ab049600d" containerName="payload-processing" containerID="cri-o://b93cf9151c91d8304a7d5c0d225adb152780633d4c1e6e2cfd767f3576f18ae4" gracePeriod=25
Nov 02 08:08:22.521994 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.521953    4898 generic.go:356] "Generic (PLEG): container finished" podID="486dad79-0ce8-42ce-83fa-746ab049600d" containerID="b93cf9151c91d8304a7d5c0d225adb152780633d4c1e6e2cfd767f3576f18ae4" exitCode=0
Nov 02 08:08:22.522573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.522085    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45" event={"ID":"486dad79-0ce8-42ce-83fa-746ab049600d","Type":"ContainerDied","Data":"b93cf9151c91d8304a7d5c0d225adb152780633d4c1e6e2cfd767f3576f18ae4"}
Nov 02 08:08:22.591669 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.591639    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:08:22.700399 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.700363    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-kube-api-access-4ndc9\" (UniqueName: \"kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-kube-api-access-4ndc9\") pod \"486dad79-0ce8-42ce-83fa-746ab049600d\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") "
Nov 02 08:08:22.700579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.700418    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-provider-credentials\") pod \"486dad79-0ce8-42ce-83fa-746ab049600d\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") "
Nov 02 08:08:22.700579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.700566    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-plugins-config-volume\") pod \"486dad79-0ce8-42ce-83fa-746ab049600d\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") "
Nov 02 08:08:22.700663 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.700611    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-routing-overlay-volume\") pod \"486dad79-0ce8-42ce-83fa-746ab049600d\" (UID: \"486dad79-0ce8-42ce-83fa-746ab049600d\") "
Nov 02 08:08:22.701102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.701067    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-plugins-config-volume" pod "486dad79-0ce8-42ce-83fa-746ab049600d" (UID: "486dad79-0ce8-42ce-83fa-746ab049600d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:08:22.701236 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.701125    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-routing-overlay-volume" pod "486dad79-0ce8-42ce-83fa-746ab049600d" (UID: "486dad79-0ce8-42ce-83fa-746ab049600d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:08:22.701513 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.701488    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:22.701619 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.701517    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486dad79-0ce8-42ce-83fa-746ab049600d-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:22.702766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.702745    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-provider-credentials" pod "486dad79-0ce8-42ce-83fa-746ab049600d" (UID: "486dad79-0ce8-42ce-83fa-746ab049600d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:22.702971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.702951    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-kube-api-access-4ndc9" pod "486dad79-0ce8-42ce-83fa-746ab049600d" (UID: "486dad79-0ce8-42ce-83fa-746ab049600d"). InnerVolumeSpecName "kube-api-access-4ndc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:22.803445 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.803401    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:22.803445 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:22.803441    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ndc9\" (UniqueName: \"kubernetes.io/projected/486dad79-0ce8-42ce-83fa-746ab049600d-kube-api-access-4ndc9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:23.532998 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:23.532965    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"
Nov 02 08:08:23.533517 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:23.533011    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45" event={"ID":"486dad79-0ce8-42ce-83fa-746ab049600d","Type":"ContainerDied","Data":"f001e2a581494ef04e43aee48363694f3868f2e7868d32bb36a9390e12fb11cc"}
Nov 02 08:08:23.533517 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:23.533077    4898 scope.go:122] "RemoveContainer" containerID="b93cf9151c91d8304a7d5c0d225adb152780633d4c1e6e2cfd767f3576f18ae4"
Nov 02 08:08:23.588057 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:23.588024    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"]
Nov 02 08:08:23.592704 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:23.592670    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-7dd5ffc45"]
Nov 02 08:08:25.468412 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:25.468365    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486dad79-0ce8-42ce-83fa-746ab049600d" path="/var/lib/kubelet/pods/486dad79-0ce8-42ce-83fa-746ab049600d/volumes"
Nov 02 08:08:26.496764 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:26.496705    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f" podUID="25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9" containerName="payload-processing" containerID="cri-o://e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785" gracePeriod=25
Nov 02 08:08:26.807842 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:26.807811    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:08:26.962699 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:26.962654    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-kube-api-access-9qdbs\" (UniqueName: \"kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-kube-api-access-9qdbs\") pod \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") "
Nov 02 08:08:26.962939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:26.962764    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-routing-overlay-volume\") pod \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") "
Nov 02 08:08:26.962939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:26.962842    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-plugins-config-volume\") pod \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") "
Nov 02 08:08:26.962939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:26.962884    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-provider-credentials\") pod \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\" (UID: \"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9\") "
Nov 02 08:08:26.963289 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:26.963258    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-routing-overlay-volume" pod "25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9" (UID: "25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:08:26.963441 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:26.963266    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-plugins-config-volume" pod "25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9" (UID: "25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:08:26.963848 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:26.963822    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:26.963967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:26.963855    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:26.965530 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:26.965493    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-kube-api-access-9qdbs" pod "25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9" (UID: "25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9"). InnerVolumeSpecName "kube-api-access-9qdbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:26.965530 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:26.965522    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-provider-credentials" pod "25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9" (UID: "25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:27.066134 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:27.066045    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:27.066134 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:27.066086    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qdbs\" (UniqueName: \"kubernetes.io/projected/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9-kube-api-access-9qdbs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:27.581752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:27.581709    4898 generic.go:356] "Generic (PLEG): container finished" podID="25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9" containerID="e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785" exitCode=0
Nov 02 08:08:27.582314 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:27.581770    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"
Nov 02 08:08:27.582314 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:27.582015    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f" event={"ID":"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9","Type":"ContainerDied","Data":"e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785"}
Nov 02 08:08:27.582314 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:27.582070    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f" event={"ID":"25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9","Type":"ContainerDied","Data":"8e750c08775fca541806c2997c50fd8904f5c4d4b69f1e0aa13f0af9025b382f"}
Nov 02 08:08:27.582314 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:27.582101    4898 scope.go:122] "RemoveContainer" containerID="e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785"
Nov 02 08:08:27.614932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:27.614864    4898 scope.go:122] "RemoveContainer" containerID="e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785"
Nov 02 08:08:27.615569 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:08:27.615542    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785\": container with ID starting with e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785 not found: ID does not exist" containerID="e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785"
Nov 02 08:08:27.615676 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:27.615579    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785"} err="failed to get container status \"e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785\": rpc error: code = NotFound desc = could not find container \"e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785\": container with ID starting with e268c17a6e04b6d74cb9eb85de8624f2baefe9fe8911eed33cd7fc7161f74785 not found: ID does not exist"
Nov 02 08:08:27.650149 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:27.650112    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"]
Nov 02 08:08:27.661805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:27.661770    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-54fdgpz6f"]
Nov 02 08:08:29.472477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:29.472433    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9" path="/var/lib/kubelet/pods/25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9/volumes"
Nov 02 08:08:31.466074 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.466029    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"]
Nov 02 08:08:31.467828 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.467795    4898 cpu_manager.go:414] "containerMap: removing container" podUID="486dad79-0ce8-42ce-83fa-746ab049600d" containerName="payload-processing"
Nov 02 08:08:31.469217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.469194    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486dad79-0ce8-42ce-83fa-746ab049600d" containerName="payload-processing"
Nov 02 08:08:31.469538 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.469519    4898 cpu_manager.go:414] "containerMap: removing container" podUID="25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9" containerName="payload-processing"
Nov 02 08:08:31.469665 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.469648    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9" containerName="payload-processing"
Nov 02 08:08:31.469813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.469796    4898 cpu_manager.go:414] "containerMap: removing container" podUID="984f0c6e-8b5d-4599-bc5a-8414b26d21c1" containerName="revoke-keys"
Nov 02 08:08:31.469982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.469965    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984f0c6e-8b5d-4599-bc5a-8414b26d21c1" containerName="revoke-keys"
Nov 02 08:08:31.470336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.470317    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="486dad79-0ce8-42ce-83fa-746ab049600d" containerName="payload-processing"
Nov 02 08:08:31.470507 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.470492    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a930b7-1dfd-4c88-a6b3-6767d7c7e8d9" containerName="payload-processing"
Nov 02 08:08:31.470672 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.470655    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="984f0c6e-8b5d-4599-bc5a-8414b26d21c1" containerName="revoke-keys"
Nov 02 08:08:31.486151 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.486117    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.517756 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.517727    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c712b1\""
Nov 02 08:08:31.518139 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.518122    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"kube-root-ca.crt\""
Nov 02 08:08:31.518409 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.518381    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"payload-processing-external-model-e2e--50b787fd-dockercfg-bpttw\""
Nov 02 08:08:31.518575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.518491    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"agc-cred-ea41b045e996\""
Nov 02 08:08:31.518674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.518608    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"openshift-service-ca.crt\""
Nov 02 08:08:31.518927 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.518875    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"routing-overlay\""
Nov 02 08:08:31.520284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.520257    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"]
Nov 02 08:08:31.698864 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.693322    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.698864 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.693490    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqg5\" (UniqueName: \"kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-kube-api-access-9xqg5\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.698864 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.693609    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.698864 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.693785    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.795421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.795313    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.795421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.795399    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqg5\" (UniqueName: \"kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-kube-api-access-9xqg5\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.795680 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.795497    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.795680 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.795648    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.796302 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.796278    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.796490 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.796462    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.798162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.798133    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.819057 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.819024    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqg5\" (UniqueName: \"kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-kube-api-access-9xqg5\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:31.823921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:31.823879    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:32.080502 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.080418    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"]
Nov 02 08:08:32.086123 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.086102    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.099287 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.099258    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.099484 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.099310    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.099484 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.099415    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w54vh\" (UniqueName: \"kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-kube-api-access-w54vh\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.099578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.099557    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.102760 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.102730    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c712b1\"/\"agc-cred-681c2d8b4aa8\""
Nov 02 08:08:32.105337 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.105313    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"]
Nov 02 08:08:32.143578 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:08:32.142592    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8432e0_70db_4939_8717_de40976fcc25.slice/crio-37fcc607e52e95cafea7863cbc911ca87b300d99bd022e1303d4fcbd2147f0f6 WatchSource:0}: Error finding container 37fcc607e52e95cafea7863cbc911ca87b300d99bd022e1303d4fcbd2147f0f6: Status 404 returned error can't find the container with id 37fcc607e52e95cafea7863cbc911ca87b300d99bd022e1303d4fcbd2147f0f6
Nov 02 08:08:32.143578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.143409    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"]
Nov 02 08:08:32.205482 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.205434    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w54vh\" (UniqueName: \"kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-kube-api-access-w54vh\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.205937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.205906    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.206055 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.205988    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.206125 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.206050    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.206958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.206825    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.207754 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.207726    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.210275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.210224    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.230817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.230779    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w54vh\" (UniqueName: \"kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-kube-api-access-w54vh\") pod \"payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") " pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.411412 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.411377    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:32.706604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.706509    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz" event={"ID":"bb8432e0-70db-4939-8717-de40976fcc25","Type":"ContainerStarted","Data":"cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329"}
Nov 02 08:08:32.706604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.706582    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz" event={"ID":"bb8432e0-70db-4939-8717-de40976fcc25","Type":"ContainerStarted","Data":"37fcc607e52e95cafea7863cbc911ca87b300d99bd022e1303d4fcbd2147f0f6"}
Nov 02 08:08:32.707056 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.706723    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:32.723192 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:08:32.723165    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd38daae_cf3a_47cd_9055_f60195e3fa92.slice/crio-dada9bfe3ba1e5db78321714a6240a52c96c412495516c1fd3737ea3f63ae7bf WatchSource:0}: Error finding container dada9bfe3ba1e5db78321714a6240a52c96c412495516c1fd3737ea3f63ae7bf: Status 404 returned error can't find the container with id dada9bfe3ba1e5db78321714a6240a52c96c412495516c1fd3737ea3f63ae7bf
Nov 02 08:08:32.724869 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.724846    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"]
Nov 02 08:08:32.751256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:32.751208    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz" podStartSLOduration=1.7511942409999999 podStartE2EDuration="1.751194241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:08:32.74741669 +0000 UTC m=+2663.437743674" watchObservedRunningTime="2026-11-02 08:08:32.751194241 +0000 UTC m=+2663.441521260"
Nov 02 08:08:33.717942 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:33.717902    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p" event={"ID":"cd38daae-cf3a-47cd-9055-f60195e3fa92","Type":"ContainerStarted","Data":"02f29ed590760cefbd91568c91e71c0cd29371b3663d6011210089d730f1d928"}
Nov 02 08:08:33.718451 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:33.717971    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p" event={"ID":"cd38daae-cf3a-47cd-9055-f60195e3fa92","Type":"ContainerStarted","Data":"dada9bfe3ba1e5db78321714a6240a52c96c412495516c1fd3737ea3f63ae7bf"}
Nov 02 08:08:33.718451 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:33.718030    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:33.762926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:33.762850    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p" podStartSLOduration=1.762834499 podStartE2EDuration="1.762834499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:08:33.760536708 +0000 UTC m=+2664.450863692" watchObservedRunningTime="2026-11-02 08:08:33.762834499 +0000 UTC m=+2664.453161463"
Nov 02 08:08:43.721581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:43.721549    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:44.730164 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:44.730129    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:08:44.809229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:44.809182    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"]
Nov 02 08:08:44.994062 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:08:44.993979    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:08:46.155637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.155588    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"]
Nov 02 08:08:46.160964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.160931    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:46.181962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.181929    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"]
Nov 02 08:08:46.186938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.184939    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:08:46.186938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.184958    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:08:46.186938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.184995    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:08:46.186938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.185035    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:08:46.337762 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.337722    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xggkz\" (UniqueName: \"kubernetes.io/projected/803fbfc1-d231-44be-b5bc-2c021dfd1940-kube-api-access-xggkz\") pod \"maas-api-revoke-sub-021a6ddc-4jmlm\" (UID: \"803fbfc1-d231-44be-b5bc-2c021dfd1940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:46.338057 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.338032    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021a6ddc-4jmlm\" (UID: \"803fbfc1-d231-44be-b5bc-2c021dfd1940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:46.338183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.338165    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021a6ddc-4jmlm\" (UID: \"803fbfc1-d231-44be-b5bc-2c021dfd1940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:46.439321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.439222    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021a6ddc-4jmlm\" (UID: \"803fbfc1-d231-44be-b5bc-2c021dfd1940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:46.439513 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.439353    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xggkz\" (UniqueName: \"kubernetes.io/projected/803fbfc1-d231-44be-b5bc-2c021dfd1940-kube-api-access-xggkz\") pod \"maas-api-revoke-sub-021a6ddc-4jmlm\" (UID: \"803fbfc1-d231-44be-b5bc-2c021dfd1940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:46.439587 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.439559    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021a6ddc-4jmlm\" (UID: \"803fbfc1-d231-44be-b5bc-2c021dfd1940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:46.439983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.439961    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021a6ddc-4jmlm\" (UID: \"803fbfc1-d231-44be-b5bc-2c021dfd1940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:46.442111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.442080    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021a6ddc-4jmlm\" (UID: \"803fbfc1-d231-44be-b5bc-2c021dfd1940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:46.463397 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.463363    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xggkz\" (UniqueName: \"kubernetes.io/projected/803fbfc1-d231-44be-b5bc-2c021dfd1940-kube-api-access-xggkz\") pod \"maas-api-revoke-sub-021a6ddc-4jmlm\" (UID: \"803fbfc1-d231-44be-b5bc-2c021dfd1940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:46.484105 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.484076    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:46.817069 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:08:46.817036    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803fbfc1_d231_44be_b5bc_2c021dfd1940.slice/crio-e8abe76fc970730386325552f5dd7989a1888f9ec73f7a3ac1cdf79aedfb8c7f WatchSource:0}: Error finding container e8abe76fc970730386325552f5dd7989a1888f9ec73f7a3ac1cdf79aedfb8c7f: Status 404 returned error can't find the container with id e8abe76fc970730386325552f5dd7989a1888f9ec73f7a3ac1cdf79aedfb8c7f
Nov 02 08:08:46.819417 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.819392    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"]
Nov 02 08:08:46.856723 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:46.856687    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm" event={"ID":"803fbfc1-d231-44be-b5bc-2c021dfd1940","Type":"ContainerStarted","Data":"e8abe76fc970730386325552f5dd7989a1888f9ec73f7a3ac1cdf79aedfb8c7f"}
Nov 02 08:08:47.868224 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:47.868187    4898 generic.go:356] "Generic (PLEG): container finished" podID="803fbfc1-d231-44be-b5bc-2c021dfd1940" containerID="0a68323103298ef27517f4d52563174b3bcfac83eb9c73c81c40b71daeb0bf03" exitCode=0
Nov 02 08:08:47.868632 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:47.868265    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm" event={"ID":"803fbfc1-d231-44be-b5bc-2c021dfd1940","Type":"ContainerDied","Data":"0a68323103298ef27517f4d52563174b3bcfac83eb9c73c81c40b71daeb0bf03"}
Nov 02 08:08:49.053004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.052968    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:49.081833 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.081790    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-openshift-ca\") pod \"803fbfc1-d231-44be-b5bc-2c021dfd1940\" (UID: \"803fbfc1-d231-44be-b5bc-2c021dfd1940\") "
Nov 02 08:08:49.082024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.081839    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803fbfc1-d231-44be-b5bc-2c021dfd1940-kube-api-access-xggkz\" (UniqueName: \"kubernetes.io/projected/803fbfc1-d231-44be-b5bc-2c021dfd1940-kube-api-access-xggkz\") pod \"803fbfc1-d231-44be-b5bc-2c021dfd1940\" (UID: \"803fbfc1-d231-44be-b5bc-2c021dfd1940\") "
Nov 02 08:08:49.082172 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.082150    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-xks-ca\") pod \"803fbfc1-d231-44be-b5bc-2c021dfd1940\" (UID: \"803fbfc1-d231-44be-b5bc-2c021dfd1940\") "
Nov 02 08:08:49.082282 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.082249    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-openshift-ca" pod "803fbfc1-d231-44be-b5bc-2c021dfd1940" (UID: "803fbfc1-d231-44be-b5bc-2c021dfd1940"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:08:49.082618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.082595    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:49.084109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.084085    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803fbfc1-d231-44be-b5bc-2c021dfd1940-kube-api-access-xggkz" pod "803fbfc1-d231-44be-b5bc-2c021dfd1940" (UID: "803fbfc1-d231-44be-b5bc-2c021dfd1940"). InnerVolumeSpecName "kube-api-access-xggkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:49.084235 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.084150    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-xks-ca" pod "803fbfc1-d231-44be-b5bc-2c021dfd1940" (UID: "803fbfc1-d231-44be-b5bc-2c021dfd1940"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:08:49.183508 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.183470    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xggkz\" (UniqueName: \"kubernetes.io/projected/803fbfc1-d231-44be-b5bc-2c021dfd1940-kube-api-access-xggkz\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:49.183508 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.183507    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803fbfc1-d231-44be-b5bc-2c021dfd1940-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:49.810561 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.810467    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz" podUID="bb8432e0-70db-4939-8717-de40976fcc25" containerName="payload-processing" containerID="cri-o://cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329" gracePeriod=25
Nov 02 08:08:49.888527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.888496    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"
Nov 02 08:08:49.888527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.888507    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm" event={"ID":"803fbfc1-d231-44be-b5bc-2c021dfd1940","Type":"ContainerDied","Data":"e8abe76fc970730386325552f5dd7989a1888f9ec73f7a3ac1cdf79aedfb8c7f"}
Nov 02 08:08:49.888750 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:49.888575    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8abe76fc970730386325552f5dd7989a1888f9ec73f7a3ac1cdf79aedfb8c7f"
Nov 02 08:08:50.105178 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.105147    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:50.201389 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.201348    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-plugins-config-volume\") pod \"bb8432e0-70db-4939-8717-de40976fcc25\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") "
Nov 02 08:08:50.201592 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.201559    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-kube-api-access-9xqg5\" (UniqueName: \"kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-kube-api-access-9xqg5\") pod \"bb8432e0-70db-4939-8717-de40976fcc25\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") "
Nov 02 08:08:50.201759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.201721    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-routing-overlay-volume\") pod \"bb8432e0-70db-4939-8717-de40976fcc25\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") "
Nov 02 08:08:50.201863 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.201782    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-provider-credentials\") pod \"bb8432e0-70db-4939-8717-de40976fcc25\" (UID: \"bb8432e0-70db-4939-8717-de40976fcc25\") "
Nov 02 08:08:50.202050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.201779    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-plugins-config-volume" pod "bb8432e0-70db-4939-8717-de40976fcc25" (UID: "bb8432e0-70db-4939-8717-de40976fcc25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:08:50.202206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.202178    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-routing-overlay-volume" pod "bb8432e0-70db-4939-8717-de40976fcc25" (UID: "bb8432e0-70db-4939-8717-de40976fcc25"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:08:50.202469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.202451    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:50.202592 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.202569    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8432e0-70db-4939-8717-de40976fcc25-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:50.204225 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.204194    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-provider-credentials" pod "bb8432e0-70db-4939-8717-de40976fcc25" (UID: "bb8432e0-70db-4939-8717-de40976fcc25"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:50.204330 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.204312    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-kube-api-access-9xqg5" pod "bb8432e0-70db-4939-8717-de40976fcc25" (UID: "bb8432e0-70db-4939-8717-de40976fcc25"). InnerVolumeSpecName "kube-api-access-9xqg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:08:50.303562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.303525    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xqg5\" (UniqueName: \"kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-kube-api-access-9xqg5\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:50.303562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.303558    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb8432e0-70db-4939-8717-de40976fcc25-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:08:50.901252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.901212    4898 generic.go:356] "Generic (PLEG): container finished" podID="bb8432e0-70db-4939-8717-de40976fcc25" containerID="cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329" exitCode=0
Nov 02 08:08:50.901472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.901296    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"
Nov 02 08:08:50.901472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.901307    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz" event={"ID":"bb8432e0-70db-4939-8717-de40976fcc25","Type":"ContainerDied","Data":"cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329"}
Nov 02 08:08:50.901472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.901413    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz" event={"ID":"bb8432e0-70db-4939-8717-de40976fcc25","Type":"ContainerDied","Data":"37fcc607e52e95cafea7863cbc911ca87b300d99bd022e1303d4fcbd2147f0f6"}
Nov 02 08:08:50.901472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.901443    4898 scope.go:122] "RemoveContainer" containerID="cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329"
Nov 02 08:08:50.915880 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.915858    4898 scope.go:122] "RemoveContainer" containerID="cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329"
Nov 02 08:08:50.916929 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:08:50.916212    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329\": container with ID starting with cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329 not found: ID does not exist" containerID="cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329"
Nov 02 08:08:50.916929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.916247    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329"} err="failed to get container status \"cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329\": rpc error: code = NotFound desc = could not find container \"cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329\": container with ID starting with cb538d70ecf9a0f07ef0510d7cedd79b4d88925168eeb6c8ac4c317500ffb329 not found: ID does not exist"
Nov 02 08:08:50.958111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.958077    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"]
Nov 02 08:08:50.971229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:50.971197    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5df8wsqbz"]
Nov 02 08:08:51.467209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:51.467122    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb8432e0-70db-4939-8717-de40976fcc25" path="/var/lib/kubelet/pods/bb8432e0-70db-4939-8717-de40976fcc25/volumes"
Nov 02 08:08:58.204657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.204615    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"]
Nov 02 08:08:58.205816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.205795    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bb8432e0-70db-4939-8717-de40976fcc25" containerName="payload-processing"
Nov 02 08:08:58.205876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.205820    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8432e0-70db-4939-8717-de40976fcc25" containerName="payload-processing"
Nov 02 08:08:58.205876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.205836    4898 cpu_manager.go:414] "containerMap: removing container" podUID="803fbfc1-d231-44be-b5bc-2c021dfd1940" containerName="revoke-keys"
Nov 02 08:08:58.205876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.205844    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803fbfc1-d231-44be-b5bc-2c021dfd1940" containerName="revoke-keys"
Nov 02 08:08:58.206069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.206033    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8432e0-70db-4939-8717-de40976fcc25" containerName="payload-processing"
Nov 02 08:08:58.206069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.206044    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="803fbfc1-d231-44be-b5bc-2c021dfd1940" containerName="revoke-keys"
Nov 02 08:08:58.211652 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.211630    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:08:58.226880 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.226853    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"]
Nov 02 08:08:58.235109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.235082    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:08:58.235308 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.235214    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:08:58.235411 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.235368    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:08:58.238847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.238810    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:08:58.329427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.329383    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgngn\" (UniqueName: \"kubernetes.io/projected/5e954a21-92cb-45a1-ba7f-e3f079ad73be-kube-api-access-dgngn\") pod \"maas-api-revoke-sub-3d52e3ac-nzvqm\" (UID: \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:08:58.329618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.329580    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d52e3ac-nzvqm\" (UID: \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:08:58.329675 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.329651    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d52e3ac-nzvqm\" (UID: \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:08:58.430677 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.430633    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d52e3ac-nzvqm\" (UID: \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:08:58.430929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.430838    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgngn\" (UniqueName: \"kubernetes.io/projected/5e954a21-92cb-45a1-ba7f-e3f079ad73be-kube-api-access-dgngn\") pod \"maas-api-revoke-sub-3d52e3ac-nzvqm\" (UID: \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:08:58.431043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.431023    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d52e3ac-nzvqm\" (UID: \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:08:58.431745 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.431722    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d52e3ac-nzvqm\" (UID: \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:08:58.433178 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.433150    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d52e3ac-nzvqm\" (UID: \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:08:58.454922 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.454831    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgngn\" (UniqueName: \"kubernetes.io/projected/5e954a21-92cb-45a1-ba7f-e3f079ad73be-kube-api-access-dgngn\") pod \"maas-api-revoke-sub-3d52e3ac-nzvqm\" (UID: \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:08:58.529958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.529885    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:08:58.852980 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:08:58.852949    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e954a21_92cb_45a1_ba7f_e3f079ad73be.slice/crio-465e3e2f265f0c2ca9b5801d78692cd8c06d0394377726c0615e1dc530e31dae WatchSource:0}: Error finding container 465e3e2f265f0c2ca9b5801d78692cd8c06d0394377726c0615e1dc530e31dae: Status 404 returned error can't find the container with id 465e3e2f265f0c2ca9b5801d78692cd8c06d0394377726c0615e1dc530e31dae
Nov 02 08:08:58.853501 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.853468    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"]
Nov 02 08:08:58.987385 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:58.987348    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm" event={"ID":"5e954a21-92cb-45a1-ba7f-e3f079ad73be","Type":"ContainerStarted","Data":"465e3e2f265f0c2ca9b5801d78692cd8c06d0394377726c0615e1dc530e31dae"}
Nov 02 08:08:59.998311 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:59.998274    4898 generic.go:356] "Generic (PLEG): container finished" podID="5e954a21-92cb-45a1-ba7f-e3f079ad73be" containerID="9d81230def02a4f54da680b0cc7ac91ef6aea86239dc039052c24e9e28496035" exitCode=0
Nov 02 08:08:59.998743 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:08:59.998362    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm" event={"ID":"5e954a21-92cb-45a1-ba7f-e3f079ad73be","Type":"ContainerDied","Data":"9d81230def02a4f54da680b0cc7ac91ef6aea86239dc039052c24e9e28496035"}
Nov 02 08:09:01.180209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:01.180174    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:09:01.275006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:01.274907    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-xks-ca\") pod \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\" (UID: \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\") "
Nov 02 08:09:01.275361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:01.275328    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-openshift-ca\") pod \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\" (UID: \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\") "
Nov 02 08:09:01.276020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:01.275999    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e954a21-92cb-45a1-ba7f-e3f079ad73be-kube-api-access-dgngn\" (UniqueName: \"kubernetes.io/projected/5e954a21-92cb-45a1-ba7f-e3f079ad73be-kube-api-access-dgngn\") pod \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\" (UID: \"5e954a21-92cb-45a1-ba7f-e3f079ad73be\") "
Nov 02 08:09:01.278517 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:01.275822    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-openshift-ca" pod "5e954a21-92cb-45a1-ba7f-e3f079ad73be" (UID: "5e954a21-92cb-45a1-ba7f-e3f079ad73be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:01.281379 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:01.281354    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e954a21-92cb-45a1-ba7f-e3f079ad73be-kube-api-access-dgngn" pod "5e954a21-92cb-45a1-ba7f-e3f079ad73be" (UID: "5e954a21-92cb-45a1-ba7f-e3f079ad73be"). InnerVolumeSpecName "kube-api-access-dgngn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:01.286050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:01.286025    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-xks-ca" pod "5e954a21-92cb-45a1-ba7f-e3f079ad73be" (UID: "5e954a21-92cb-45a1-ba7f-e3f079ad73be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:09:01.379552 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:01.379509    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:01.379552 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:01.379555    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e954a21-92cb-45a1-ba7f-e3f079ad73be-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:01.379797 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:01.379574    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgngn\" (UniqueName: \"kubernetes.io/projected/5e954a21-92cb-45a1-ba7f-e3f079ad73be-kube-api-access-dgngn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:02.017717 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:02.017684    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"
Nov 02 08:09:02.017923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:02.017718    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm" event={"ID":"5e954a21-92cb-45a1-ba7f-e3f079ad73be","Type":"ContainerDied","Data":"465e3e2f265f0c2ca9b5801d78692cd8c06d0394377726c0615e1dc530e31dae"}
Nov 02 08:09:02.017923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:02.017766    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="465e3e2f265f0c2ca9b5801d78692cd8c06d0394377726c0615e1dc530e31dae"
Nov 02 08:09:09.304973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.304936    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"]
Nov 02 08:09:09.305703 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.305680    4898 cpu_manager.go:414] "containerMap: removing container" podUID="5e954a21-92cb-45a1-ba7f-e3f079ad73be" containerName="revoke-keys"
Nov 02 08:09:09.305703 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.305699    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e954a21-92cb-45a1-ba7f-e3f079ad73be" containerName="revoke-keys"
Nov 02 08:09:09.305869 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.305837    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e954a21-92cb-45a1-ba7f-e3f079ad73be" containerName="revoke-keys"
Nov 02 08:09:09.309716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.309699    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:09.326320 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.326291    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:09:09.330645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.330620    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"]
Nov 02 08:09:09.337694 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.337667    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:09:09.337853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.337725    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:09:09.337853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.337832    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:09:09.408509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.408451    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2725r\" (UniqueName: \"kubernetes.io/projected/b82c790c-ea55-4cea-bceb-ccd4eac21567-kube-api-access-2725r\") pod \"maas-api-revoke-sub-270f250e-687qg\" (UID: \"b82c790c-ea55-4cea-bceb-ccd4eac21567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:09.408698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.408627    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-270f250e-687qg\" (UID: \"b82c790c-ea55-4cea-bceb-ccd4eac21567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:09.408698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.408678    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-270f250e-687qg\" (UID: \"b82c790c-ea55-4cea-bceb-ccd4eac21567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:09.512599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.512558    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2725r\" (UniqueName: \"kubernetes.io/projected/b82c790c-ea55-4cea-bceb-ccd4eac21567-kube-api-access-2725r\") pod \"maas-api-revoke-sub-270f250e-687qg\" (UID: \"b82c790c-ea55-4cea-bceb-ccd4eac21567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:09.516076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.516049    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-270f250e-687qg\" (UID: \"b82c790c-ea55-4cea-bceb-ccd4eac21567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:09.516287 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.516265    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-270f250e-687qg\" (UID: \"b82c790c-ea55-4cea-bceb-ccd4eac21567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:09.530539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.530515    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:09:09.530669 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.530634    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:09:09.536853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.536824    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-270f250e-687qg\" (UID: \"b82c790c-ea55-4cea-bceb-ccd4eac21567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:09.542272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.542248    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-270f250e-687qg\" (UID: \"b82c790c-ea55-4cea-bceb-ccd4eac21567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:09.550687 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.550657    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:09:09.567655 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.567584    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2725r\" (UniqueName: \"kubernetes.io/projected/b82c790c-ea55-4cea-bceb-ccd4eac21567-kube-api-access-2725r\") pod \"maas-api-revoke-sub-270f250e-687qg\" (UID: \"b82c790c-ea55-4cea-bceb-ccd4eac21567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:09.641331 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.641292    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:09:09.661910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:09.646005    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:10.311915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:10.300555    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"]
Nov 02 08:09:10.408987 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:10.408960    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:09:10.410022 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:10.409995    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:09:11.007804 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:11.007711    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"]
Nov 02 08:09:11.126130 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:11.126091    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg" event={"ID":"b82c790c-ea55-4cea-bceb-ccd4eac21567","Type":"ContainerStarted","Data":"1d1247b2cbc987e78cc9dca0f2fbfcb7a5121e85a6b83bb7bd3cf649c3865d9b"}
Nov 02 08:09:11.126315 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:11.126151    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg" event={"ID":"b82c790c-ea55-4cea-bceb-ccd4eac21567","Type":"ContainerStarted","Data":"a31a2f6e336b6878b65b39dae3e2ac3e6645483e8ba2dcbd41195a6f006e1e80"}
Nov 02 08:09:11.171052 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:11.170986    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg" podStartSLOduration=2.170966918 podStartE2EDuration="2.170966918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:09:11.166232794 +0000 UTC m=+2701.856559759" watchObservedRunningTime="2026-11-02 08:09:11.170966918 +0000 UTC m=+2701.861293887"
Nov 02 08:09:12.149175 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:12.149134    4898 generic.go:356] "Generic (PLEG): container finished" podID="b82c790c-ea55-4cea-bceb-ccd4eac21567" containerID="1d1247b2cbc987e78cc9dca0f2fbfcb7a5121e85a6b83bb7bd3cf649c3865d9b" exitCode=0
Nov 02 08:09:12.149677 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:12.149217    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg" event={"ID":"b82c790c-ea55-4cea-bceb-ccd4eac21567","Type":"ContainerDied","Data":"1d1247b2cbc987e78cc9dca0f2fbfcb7a5121e85a6b83bb7bd3cf649c3865d9b"}
Nov 02 08:09:13.360921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:13.360862    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:13.471058 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:13.452527    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-openshift-ca\") pod \"b82c790c-ea55-4cea-bceb-ccd4eac21567\" (UID: \"b82c790c-ea55-4cea-bceb-ccd4eac21567\") "
Nov 02 08:09:13.471058 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:13.452787    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-xks-ca\") pod \"b82c790c-ea55-4cea-bceb-ccd4eac21567\" (UID: \"b82c790c-ea55-4cea-bceb-ccd4eac21567\") "
Nov 02 08:09:13.471058 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:13.452860    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82c790c-ea55-4cea-bceb-ccd4eac21567-kube-api-access-2725r\" (UniqueName: \"kubernetes.io/projected/b82c790c-ea55-4cea-bceb-ccd4eac21567-kube-api-access-2725r\") pod \"b82c790c-ea55-4cea-bceb-ccd4eac21567\" (UID: \"b82c790c-ea55-4cea-bceb-ccd4eac21567\") "
Nov 02 08:09:13.480925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:13.478665    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-openshift-ca" pod "b82c790c-ea55-4cea-bceb-ccd4eac21567" (UID: "b82c790c-ea55-4cea-bceb-ccd4eac21567"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:13.505622 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:13.505584    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82c790c-ea55-4cea-bceb-ccd4eac21567-kube-api-access-2725r" pod "b82c790c-ea55-4cea-bceb-ccd4eac21567" (UID: "b82c790c-ea55-4cea-bceb-ccd4eac21567"). InnerVolumeSpecName "kube-api-access-2725r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:13.513916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:13.513867    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-xks-ca" pod "b82c790c-ea55-4cea-bceb-ccd4eac21567" (UID: "b82c790c-ea55-4cea-bceb-ccd4eac21567"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:09:13.554720 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:13.554681    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:13.554720 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:13.554716    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2725r\" (UniqueName: \"kubernetes.io/projected/b82c790c-ea55-4cea-bceb-ccd4eac21567-kube-api-access-2725r\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:13.554932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:13.554736    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82c790c-ea55-4cea-bceb-ccd4eac21567-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:14.171058 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:14.171022    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"
Nov 02 08:09:14.171277 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:14.171051    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg" event={"ID":"b82c790c-ea55-4cea-bceb-ccd4eac21567","Type":"ContainerDied","Data":"a31a2f6e336b6878b65b39dae3e2ac3e6645483e8ba2dcbd41195a6f006e1e80"}
Nov 02 08:09:14.171277 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:14.171101    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31a2f6e336b6878b65b39dae3e2ac3e6645483e8ba2dcbd41195a6f006e1e80"
Nov 02 08:09:14.990409 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:09:14.990372    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:09:16.008971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.008928    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2" podUID="9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce" containerName="payload-processing" containerID="cri-o://3d0c3e79097d5541cd515946f46d9d9986c22d60f9672126f7d9d351810aefab" gracePeriod=25
Nov 02 08:09:16.196169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.196045    4898 generic.go:356] "Generic (PLEG): container finished" podID="9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce" containerID="3d0c3e79097d5541cd515946f46d9d9986c22d60f9672126f7d9d351810aefab" exitCode=0
Nov 02 08:09:16.196169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.196134    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2" event={"ID":"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce","Type":"ContainerDied","Data":"3d0c3e79097d5541cd515946f46d9d9986c22d60f9672126f7d9d351810aefab"}
Nov 02 08:09:16.294943 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.294872    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:09:16.391407 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.391367    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-kube-api-access-lqtn9\" (UniqueName: \"kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-kube-api-access-lqtn9\") pod \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") "
Nov 02 08:09:16.391585 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.391537    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-provider-credentials\") pod \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") "
Nov 02 08:09:16.391682 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.391663    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-routing-overlay-volume\") pod \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") "
Nov 02 08:09:16.391810 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.391780    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-plugins-config-volume\") pod \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\" (UID: \"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce\") "
Nov 02 08:09:16.392297 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.392268    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-routing-overlay-volume" pod "9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce" (UID: "9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:16.392454 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.392275    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-plugins-config-volume" pod "9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce" (UID: "9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:16.394229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.394205    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-kube-api-access-lqtn9" pod "9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce" (UID: "9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce"). InnerVolumeSpecName "kube-api-access-lqtn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:16.394459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.394434    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-provider-credentials" pod "9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce" (UID: "9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:16.494089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.494052    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:16.494089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.494083    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqtn9\" (UniqueName: \"kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-kube-api-access-lqtn9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:16.494089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.494096    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:16.494426 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:16.494109    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:17.216274 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:17.216230    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2" event={"ID":"9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce","Type":"ContainerDied","Data":"3d0668f244ba8a4c09442ff6c233acc87b030a5ad9855a9931df7cbaa83817d5"}
Nov 02 08:09:17.216788 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:17.216339    4898 scope.go:122] "RemoveContainer" containerID="3d0c3e79097d5541cd515946f46d9d9986c22d60f9672126f7d9d351810aefab"
Nov 02 08:09:17.216788 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:17.216523    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"
Nov 02 08:09:17.292436 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:17.292404    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"]
Nov 02 08:09:17.304069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:17.304041    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-7567pr6q2"]
Nov 02 08:09:17.467627 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:17.467549    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce" path="/var/lib/kubelet/pods/9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce/volumes"
Nov 02 08:09:20.111212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.111176    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"]
Nov 02 08:09:20.111901 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.111877    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce" containerName="payload-processing"
Nov 02 08:09:20.111978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.111905    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce" containerName="payload-processing"
Nov 02 08:09:20.111978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.111971    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b82c790c-ea55-4cea-bceb-ccd4eac21567" containerName="revoke-keys"
Nov 02 08:09:20.112449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.111980    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82c790c-ea55-4cea-bceb-ccd4eac21567" containerName="revoke-keys"
Nov 02 08:09:20.112449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.112116    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82c790c-ea55-4cea-bceb-ccd4eac21567" containerName="revoke-keys"
Nov 02 08:09:20.112449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.112126    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1754b7-2c49-4c96-a50a-1beaaa2fc3ce" containerName="payload-processing"
Nov 02 08:09:20.117397 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.117374    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:20.134661 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.134632    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"]
Nov 02 08:09:20.141313 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.141287    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:09:20.141562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.141543    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:09:20.141654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.141553    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:09:20.144496 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.144475    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:09:20.147212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.147188    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nncpm\" (UniqueName: \"kubernetes.io/projected/72054409-d8c7-4201-ad57-4eb8a67ded7a-kube-api-access-nncpm\") pod \"maas-api-revoke-sub-5b222260-q25xl\" (UID: \"72054409-d8c7-4201-ad57-4eb8a67ded7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:20.147320 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.147253    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b222260-q25xl\" (UID: \"72054409-d8c7-4201-ad57-4eb8a67ded7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:20.147485 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.147457    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b222260-q25xl\" (UID: \"72054409-d8c7-4201-ad57-4eb8a67ded7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:20.248706 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.248672    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b222260-q25xl\" (UID: \"72054409-d8c7-4201-ad57-4eb8a67ded7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:20.248913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.248810    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b222260-q25xl\" (UID: \"72054409-d8c7-4201-ad57-4eb8a67ded7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:20.249040 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.249020    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nncpm\" (UniqueName: \"kubernetes.io/projected/72054409-d8c7-4201-ad57-4eb8a67ded7a-kube-api-access-nncpm\") pod \"maas-api-revoke-sub-5b222260-q25xl\" (UID: \"72054409-d8c7-4201-ad57-4eb8a67ded7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:20.249518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.249493    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b222260-q25xl\" (UID: \"72054409-d8c7-4201-ad57-4eb8a67ded7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:20.251197 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.251175    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b222260-q25xl\" (UID: \"72054409-d8c7-4201-ad57-4eb8a67ded7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:20.272730 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.272698    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nncpm\" (UniqueName: \"kubernetes.io/projected/72054409-d8c7-4201-ad57-4eb8a67ded7a-kube-api-access-nncpm\") pod \"maas-api-revoke-sub-5b222260-q25xl\" (UID: \"72054409-d8c7-4201-ad57-4eb8a67ded7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:20.433750 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.433713    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:20.757176 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:09:20.757140    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72054409_d8c7_4201_ad57_4eb8a67ded7a.slice/crio-2362e5f44d6539b5a9787e033725feffcf9709f57e5c7a5048726802e8ecde77 WatchSource:0}: Error finding container 2362e5f44d6539b5a9787e033725feffcf9709f57e5c7a5048726802e8ecde77: Status 404 returned error can't find the container with id 2362e5f44d6539b5a9787e033725feffcf9709f57e5c7a5048726802e8ecde77
Nov 02 08:09:20.758871 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:20.758845    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"]
Nov 02 08:09:21.257604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:21.257562    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl" event={"ID":"72054409-d8c7-4201-ad57-4eb8a67ded7a","Type":"ContainerStarted","Data":"637b6fb437ff275e0f2cfcd6b5fe8a9a12df293e37214b20b2a0ea6f1421e05c"}
Nov 02 08:09:21.258148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:21.257634    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl" event={"ID":"72054409-d8c7-4201-ad57-4eb8a67ded7a","Type":"ContainerStarted","Data":"2362e5f44d6539b5a9787e033725feffcf9709f57e5c7a5048726802e8ecde77"}
Nov 02 08:09:21.303575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:21.303523    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl" podStartSLOduration=1.303508665 podStartE2EDuration="1.303508665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:09:21.299389635 +0000 UTC m=+2711.989716618" watchObservedRunningTime="2026-11-02 08:09:21.303508665 +0000 UTC m=+2711.993835645"
Nov 02 08:09:22.269001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:22.268969    4898 generic.go:356] "Generic (PLEG): container finished" podID="72054409-d8c7-4201-ad57-4eb8a67ded7a" containerID="637b6fb437ff275e0f2cfcd6b5fe8a9a12df293e37214b20b2a0ea6f1421e05c" exitCode=0
Nov 02 08:09:22.269430 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:22.269033    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl" event={"ID":"72054409-d8c7-4201-ad57-4eb8a67ded7a","Type":"ContainerDied","Data":"637b6fb437ff275e0f2cfcd6b5fe8a9a12df293e37214b20b2a0ea6f1421e05c"}
Nov 02 08:09:23.464999 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:23.464958    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:23.504423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:23.504381    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-openshift-ca\") pod \"72054409-d8c7-4201-ad57-4eb8a67ded7a\" (UID: \"72054409-d8c7-4201-ad57-4eb8a67ded7a\") "
Nov 02 08:09:23.504614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:23.504518    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-xks-ca\") pod \"72054409-d8c7-4201-ad57-4eb8a67ded7a\" (UID: \"72054409-d8c7-4201-ad57-4eb8a67ded7a\") "
Nov 02 08:09:23.504614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:23.504605    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72054409-d8c7-4201-ad57-4eb8a67ded7a-kube-api-access-nncpm\" (UniqueName: \"kubernetes.io/projected/72054409-d8c7-4201-ad57-4eb8a67ded7a-kube-api-access-nncpm\") pod \"72054409-d8c7-4201-ad57-4eb8a67ded7a\" (UID: \"72054409-d8c7-4201-ad57-4eb8a67ded7a\") "
Nov 02 08:09:23.504846 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:23.504815    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-openshift-ca" pod "72054409-d8c7-4201-ad57-4eb8a67ded7a" (UID: "72054409-d8c7-4201-ad57-4eb8a67ded7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:23.505394 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:23.505366    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:23.506855 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:23.506831    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72054409-d8c7-4201-ad57-4eb8a67ded7a-kube-api-access-nncpm" pod "72054409-d8c7-4201-ad57-4eb8a67ded7a" (UID: "72054409-d8c7-4201-ad57-4eb8a67ded7a"). InnerVolumeSpecName "kube-api-access-nncpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:23.506967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:23.506879    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-xks-ca" pod "72054409-d8c7-4201-ad57-4eb8a67ded7a" (UID: "72054409-d8c7-4201-ad57-4eb8a67ded7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:09:23.606568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:23.606480    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72054409-d8c7-4201-ad57-4eb8a67ded7a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:23.606568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:23.606514    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nncpm\" (UniqueName: \"kubernetes.io/projected/72054409-d8c7-4201-ad57-4eb8a67ded7a-kube-api-access-nncpm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:24.290858 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:24.290822    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"
Nov 02 08:09:24.290858 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:24.290852    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl" event={"ID":"72054409-d8c7-4201-ad57-4eb8a67ded7a","Type":"ContainerDied","Data":"2362e5f44d6539b5a9787e033725feffcf9709f57e5c7a5048726802e8ecde77"}
Nov 02 08:09:24.291164 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:24.290919    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2362e5f44d6539b5a9787e033725feffcf9709f57e5c7a5048726802e8ecde77"
Nov 02 08:09:31.530796 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.530757    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"]
Nov 02 08:09:31.531826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.531802    4898 cpu_manager.go:414] "containerMap: removing container" podUID="72054409-d8c7-4201-ad57-4eb8a67ded7a" containerName="revoke-keys"
Nov 02 08:09:31.531978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.531828    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72054409-d8c7-4201-ad57-4eb8a67ded7a" containerName="revoke-keys"
Nov 02 08:09:31.532054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.532034    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="72054409-d8c7-4201-ad57-4eb8a67ded7a" containerName="revoke-keys"
Nov 02 08:09:31.535787 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.535766    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:31.555449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.555418    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"]
Nov 02 08:09:31.561654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.561628    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:09:31.562039 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.562022    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:09:31.562220 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.562207    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:09:31.563233 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.563216    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:09:31.679439 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.679397    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c95e5285-sl8dn\" (UID: \"741b8f34-3bac-4607-a2dc-e97073dd9582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:31.684035 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.679876    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c95e5285-sl8dn\" (UID: \"741b8f34-3bac-4607-a2dc-e97073dd9582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:31.684347 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.684326    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bg7\" (UniqueName: \"kubernetes.io/projected/741b8f34-3bac-4607-a2dc-e97073dd9582-kube-api-access-t2bg7\") pod \"maas-api-revoke-sub-c95e5285-sl8dn\" (UID: \"741b8f34-3bac-4607-a2dc-e97073dd9582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:31.786287 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.786169    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c95e5285-sl8dn\" (UID: \"741b8f34-3bac-4607-a2dc-e97073dd9582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:31.786287 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.786289    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bg7\" (UniqueName: \"kubernetes.io/projected/741b8f34-3bac-4607-a2dc-e97073dd9582-kube-api-access-t2bg7\") pod \"maas-api-revoke-sub-c95e5285-sl8dn\" (UID: \"741b8f34-3bac-4607-a2dc-e97073dd9582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:31.786574 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.786396    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c95e5285-sl8dn\" (UID: \"741b8f34-3bac-4607-a2dc-e97073dd9582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:31.787000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.786968    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c95e5285-sl8dn\" (UID: \"741b8f34-3bac-4607-a2dc-e97073dd9582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:31.788841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.788816    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c95e5285-sl8dn\" (UID: \"741b8f34-3bac-4607-a2dc-e97073dd9582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:31.816372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.816342    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bg7\" (UniqueName: \"kubernetes.io/projected/741b8f34-3bac-4607-a2dc-e97073dd9582-kube-api-access-t2bg7\") pod \"maas-api-revoke-sub-c95e5285-sl8dn\" (UID: \"741b8f34-3bac-4607-a2dc-e97073dd9582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:31.852654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:31.852621    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:32.160689 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:09:32.160650    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741b8f34_3bac_4607_a2dc_e97073dd9582.slice/crio-df778045570626932033db609e9b799d83cdf208a2340b28c624799a6d1a9429 WatchSource:0}: Error finding container df778045570626932033db609e9b799d83cdf208a2340b28c624799a6d1a9429: Status 404 returned error can't find the container with id df778045570626932033db609e9b799d83cdf208a2340b28c624799a6d1a9429
Nov 02 08:09:32.182945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:32.181976    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"]
Nov 02 08:09:32.378846 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:32.378809    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn" event={"ID":"741b8f34-3bac-4607-a2dc-e97073dd9582","Type":"ContainerStarted","Data":"df778045570626932033db609e9b799d83cdf208a2340b28c624799a6d1a9429"}
Nov 02 08:09:32.764219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:32.764112    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"]
Nov 02 08:09:33.418913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:33.413685    4898 generic.go:356] "Generic (PLEG): container finished" podID="741b8f34-3bac-4607-a2dc-e97073dd9582" containerID="ab828990841a70be5255bbf51297f2198bc5ef55b6c1e1c2be5533c44ed5b7ad" exitCode=0
Nov 02 08:09:33.418913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:33.413774    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn" event={"ID":"741b8f34-3bac-4607-a2dc-e97073dd9582","Type":"ContainerDied","Data":"ab828990841a70be5255bbf51297f2198bc5ef55b6c1e1c2be5533c44ed5b7ad"}
Nov 02 08:09:34.130848 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.130810    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"]
Nov 02 08:09:34.135477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.135458    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:34.158265 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.158233    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"]
Nov 02 08:09:34.241907 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.241853    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6097cd29-px5qz\" (UID: \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:34.242212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.242186    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6097cd29-px5qz\" (UID: \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:34.242300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.242253    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btz2d\" (UniqueName: \"kubernetes.io/projected/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-kube-api-access-btz2d\") pod \"maas-api-revoke-sub-6097cd29-px5qz\" (UID: \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:34.344122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.344082    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6097cd29-px5qz\" (UID: \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:34.344315 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.344147    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btz2d\" (UniqueName: \"kubernetes.io/projected/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-kube-api-access-btz2d\") pod \"maas-api-revoke-sub-6097cd29-px5qz\" (UID: \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:34.344315 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.344216    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6097cd29-px5qz\" (UID: \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:34.344769 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.344743    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6097cd29-px5qz\" (UID: \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:34.346596 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.346571    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6097cd29-px5qz\" (UID: \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:34.369371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.369340    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btz2d\" (UniqueName: \"kubernetes.io/projected/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-kube-api-access-btz2d\") pod \"maas-api-revoke-sub-6097cd29-px5qz\" (UID: \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:34.449765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.449732    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:34.622907 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.622854    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:34.749798 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.749711    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-openshift-ca\") pod \"741b8f34-3bac-4607-a2dc-e97073dd9582\" (UID: \"741b8f34-3bac-4607-a2dc-e97073dd9582\") "
Nov 02 08:09:34.750013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.749861    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741b8f34-3bac-4607-a2dc-e97073dd9582-kube-api-access-t2bg7\" (UniqueName: \"kubernetes.io/projected/741b8f34-3bac-4607-a2dc-e97073dd9582-kube-api-access-t2bg7\") pod \"741b8f34-3bac-4607-a2dc-e97073dd9582\" (UID: \"741b8f34-3bac-4607-a2dc-e97073dd9582\") "
Nov 02 08:09:34.750111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.750091    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-xks-ca\") pod \"741b8f34-3bac-4607-a2dc-e97073dd9582\" (UID: \"741b8f34-3bac-4607-a2dc-e97073dd9582\") "
Nov 02 08:09:34.750215 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.750192    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-openshift-ca" pod "741b8f34-3bac-4607-a2dc-e97073dd9582" (UID: "741b8f34-3bac-4607-a2dc-e97073dd9582"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:34.750706 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.750681    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:34.752226 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.752192    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-xks-ca" pod "741b8f34-3bac-4607-a2dc-e97073dd9582" (UID: "741b8f34-3bac-4607-a2dc-e97073dd9582"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:09:34.752345 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.752327    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741b8f34-3bac-4607-a2dc-e97073dd9582-kube-api-access-t2bg7" pod "741b8f34-3bac-4607-a2dc-e97073dd9582" (UID: "741b8f34-3bac-4607-a2dc-e97073dd9582"). InnerVolumeSpecName "kube-api-access-t2bg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:34.800513 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:09:34.800460    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8f11aa_420b_4c67_aeb7_2eb5a62cebb5.slice/crio-d4baae3c13cd01601d723ef2422632f6b828c3697000e9cddbc7de508dcf4bfb WatchSource:0}: Error finding container d4baae3c13cd01601d723ef2422632f6b828c3697000e9cddbc7de508dcf4bfb: Status 404 returned error can't find the container with id d4baae3c13cd01601d723ef2422632f6b828c3697000e9cddbc7de508dcf4bfb
Nov 02 08:09:34.800647 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.800572    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"]
Nov 02 08:09:34.852591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.852553    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2bg7\" (UniqueName: \"kubernetes.io/projected/741b8f34-3bac-4607-a2dc-e97073dd9582-kube-api-access-t2bg7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:34.852591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:34.852591    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741b8f34-3bac-4607-a2dc-e97073dd9582-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:35.436037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:35.435935    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz" event={"ID":"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5","Type":"ContainerStarted","Data":"5c24f975df9f60a011361d4e21014927870aecd6027fe115f4e5305391f73d67"}
Nov 02 08:09:35.436037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:35.436008    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz" event={"ID":"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5","Type":"ContainerStarted","Data":"d4baae3c13cd01601d723ef2422632f6b828c3697000e9cddbc7de508dcf4bfb"}
Nov 02 08:09:35.437681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:35.437651    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"
Nov 02 08:09:35.437816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:35.437689    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn" event={"ID":"741b8f34-3bac-4607-a2dc-e97073dd9582","Type":"ContainerDied","Data":"df778045570626932033db609e9b799d83cdf208a2340b28c624799a6d1a9429"}
Nov 02 08:09:35.437816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:35.437731    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df778045570626932033db609e9b799d83cdf208a2340b28c624799a6d1a9429"
Nov 02 08:09:35.490492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:35.490426    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz" podStartSLOduration=1.490408143 podStartE2EDuration="1.490408143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:09:35.482123484 +0000 UTC m=+2726.172450447" watchObservedRunningTime="2026-11-02 08:09:35.490408143 +0000 UTC m=+2726.180735108"
Nov 02 08:09:36.449736 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:36.449700    4898 generic.go:356] "Generic (PLEG): container finished" podID="de8f11aa-420b-4c67-aeb7-2eb5a62cebb5" containerID="5c24f975df9f60a011361d4e21014927870aecd6027fe115f4e5305391f73d67" exitCode=0
Nov 02 08:09:36.450230 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:36.449792    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz" event={"ID":"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5","Type":"ContainerDied","Data":"5c24f975df9f60a011361d4e21014927870aecd6027fe115f4e5305391f73d67"}
Nov 02 08:09:37.642430 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:37.642400    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:37.765295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:37.765198    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p" podUID="7f28770f-1e0d-484e-b0ea-874058346bb6" containerName="payload-processing" containerID="cri-o://89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d" gracePeriod=25
Nov 02 08:09:37.798247 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:37.798215    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-openshift-ca\") pod \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\" (UID: \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\") "
Nov 02 08:09:37.798452 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:37.798434    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-xks-ca\") pod \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\" (UID: \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\") "
Nov 02 08:09:37.798523 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:37.798492    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-kube-api-access-btz2d\" (UniqueName: \"kubernetes.io/projected/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-kube-api-access-btz2d\") pod \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\" (UID: \"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5\") "
Nov 02 08:09:37.798590 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:37.798531    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-openshift-ca" pod "de8f11aa-420b-4c67-aeb7-2eb5a62cebb5" (UID: "de8f11aa-420b-4c67-aeb7-2eb5a62cebb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:37.799170 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:37.799144    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:37.800535 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:37.800505    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-xks-ca" pod "de8f11aa-420b-4c67-aeb7-2eb5a62cebb5" (UID: "de8f11aa-420b-4c67-aeb7-2eb5a62cebb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:09:37.800681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:37.800656    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-kube-api-access-btz2d" pod "de8f11aa-420b-4c67-aeb7-2eb5a62cebb5" (UID: "de8f11aa-420b-4c67-aeb7-2eb5a62cebb5"). InnerVolumeSpecName "kube-api-access-btz2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:37.901001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:37.900956    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:37.901001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:37.900994    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btz2d\" (UniqueName: \"kubernetes.io/projected/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5-kube-api-access-btz2d\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:38.123766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.123736    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:09:38.210917 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.210591    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-kube-api-access-r22lc\" (UniqueName: \"kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-kube-api-access-r22lc\") pod \"7f28770f-1e0d-484e-b0ea-874058346bb6\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") "
Nov 02 08:09:38.210917 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.210659    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-provider-credentials\") pod \"7f28770f-1e0d-484e-b0ea-874058346bb6\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") "
Nov 02 08:09:38.210917 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.210819    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-routing-overlay-volume\") pod \"7f28770f-1e0d-484e-b0ea-874058346bb6\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") "
Nov 02 08:09:38.211246 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.210987    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-plugins-config-volume\") pod \"7f28770f-1e0d-484e-b0ea-874058346bb6\" (UID: \"7f28770f-1e0d-484e-b0ea-874058346bb6\") "
Nov 02 08:09:38.216112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.216073    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-routing-overlay-volume" pod "7f28770f-1e0d-484e-b0ea-874058346bb6" (UID: "7f28770f-1e0d-484e-b0ea-874058346bb6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:38.216529 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.216500    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-plugins-config-volume" pod "7f28770f-1e0d-484e-b0ea-874058346bb6" (UID: "7f28770f-1e0d-484e-b0ea-874058346bb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:38.217678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.217641    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-provider-credentials" pod "7f28770f-1e0d-484e-b0ea-874058346bb6" (UID: "7f28770f-1e0d-484e-b0ea-874058346bb6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:38.219123 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.219087    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-kube-api-access-r22lc" pod "7f28770f-1e0d-484e-b0ea-874058346bb6" (UID: "7f28770f-1e0d-484e-b0ea-874058346bb6"). InnerVolumeSpecName "kube-api-access-r22lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:38.313371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.313327    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:38.313371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.313366    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f28770f-1e0d-484e-b0ea-874058346bb6-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:38.313371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.313375    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r22lc\" (UniqueName: \"kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-kube-api-access-r22lc\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:38.313624 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.313384    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f28770f-1e0d-484e-b0ea-874058346bb6-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:38.471390 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.471350    4898 generic.go:356] "Generic (PLEG): container finished" podID="7f28770f-1e0d-484e-b0ea-874058346bb6" containerID="89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d" exitCode=0
Nov 02 08:09:38.471588 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.471422    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"
Nov 02 08:09:38.471588 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.471441    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p" event={"ID":"7f28770f-1e0d-484e-b0ea-874058346bb6","Type":"ContainerDied","Data":"89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d"}
Nov 02 08:09:38.471588 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.471513    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p" event={"ID":"7f28770f-1e0d-484e-b0ea-874058346bb6","Type":"ContainerDied","Data":"c124ce0052e78bedd2ff3c7b8841d7c1054529e3abd3d68a288f97b405d2726e"}
Nov 02 08:09:38.471588 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.471542    4898 scope.go:122] "RemoveContainer" containerID="89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d"
Nov 02 08:09:38.473367 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.473345    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"
Nov 02 08:09:38.473480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.473382    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz" event={"ID":"de8f11aa-420b-4c67-aeb7-2eb5a62cebb5","Type":"ContainerDied","Data":"d4baae3c13cd01601d723ef2422632f6b828c3697000e9cddbc7de508dcf4bfb"}
Nov 02 08:09:38.473480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.473424    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4baae3c13cd01601d723ef2422632f6b828c3697000e9cddbc7de508dcf4bfb"
Nov 02 08:09:38.484231 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.484209    4898 scope.go:122] "RemoveContainer" containerID="89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d"
Nov 02 08:09:38.484499 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:09:38.484477    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d\": container with ID starting with 89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d not found: ID does not exist" containerID="89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d"
Nov 02 08:09:38.484571 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.484509    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d"} err="failed to get container status \"89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d\": rpc error: code = NotFound desc = could not find container \"89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d\": container with ID starting with 89c51e7ee6bc5f9a136e22819d98acc8ea08f78bb9c7b2acc636e3f6009bbb4d not found: ID does not exist"
Nov 02 08:09:38.554159 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.554121    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"]
Nov 02 08:09:38.568199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:38.568157    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-7f6fpt96p"]
Nov 02 08:09:39.466816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:39.466782    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f28770f-1e0d-484e-b0ea-874058346bb6" path="/var/lib/kubelet/pods/7f28770f-1e0d-484e-b0ea-874058346bb6/volumes"
Nov 02 08:09:43.884710 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:43.884673    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"]
Nov 02 08:09:44.993288 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:09:44.993244    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:09:48.885111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:48.885063    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6" podUID="27383256-374d-49e2-bdf3-74a20ce00532" containerName="payload-processing" containerID="cri-o://1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97" gracePeriod=25
Nov 02 08:09:49.344610 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.344577    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:09:49.436689 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.436648    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-plugins-config-volume\") pod \"27383256-374d-49e2-bdf3-74a20ce00532\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") "
Nov 02 08:09:49.437103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.437074    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-routing-overlay-volume\") pod \"27383256-374d-49e2-bdf3-74a20ce00532\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") "
Nov 02 08:09:49.437805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.437781    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-kube-api-access-hs4k2\" (UniqueName: \"kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-kube-api-access-hs4k2\") pod \"27383256-374d-49e2-bdf3-74a20ce00532\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") "
Nov 02 08:09:49.437958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.437841    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-provider-credentials\" (UniqueName: \"kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-provider-credentials\") pod \"27383256-374d-49e2-bdf3-74a20ce00532\" (UID: \"27383256-374d-49e2-bdf3-74a20ce00532\") "
Nov 02 08:09:49.438019 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.437233    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-plugins-config-volume" pod "27383256-374d-49e2-bdf3-74a20ce00532" (UID: "27383256-374d-49e2-bdf3-74a20ce00532"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:49.438019 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.437678    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-routing-overlay-volume" pod "27383256-374d-49e2-bdf3-74a20ce00532" (UID: "27383256-374d-49e2-bdf3-74a20ce00532"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:09:49.439245 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.439218    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:49.439339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.439248    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27383256-374d-49e2-bdf3-74a20ce00532-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:49.441186 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.441137    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-provider-credentials" pod "27383256-374d-49e2-bdf3-74a20ce00532" (UID: "27383256-374d-49e2-bdf3-74a20ce00532"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:49.441741 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.441709    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-kube-api-access-hs4k2" pod "27383256-374d-49e2-bdf3-74a20ce00532" (UID: "27383256-374d-49e2-bdf3-74a20ce00532"). InnerVolumeSpecName "kube-api-access-hs4k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:09:49.541011 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.540976    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:49.541011 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.541007    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs4k2\" (UniqueName: \"kubernetes.io/projected/27383256-374d-49e2-bdf3-74a20ce00532-kube-api-access-hs4k2\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:09:49.584481 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.584437    4898 generic.go:356] "Generic (PLEG): container finished" podID="27383256-374d-49e2-bdf3-74a20ce00532" containerID="1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97" exitCode=0
Nov 02 08:09:49.584693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.584486    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"
Nov 02 08:09:49.584693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.584519    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6" event={"ID":"27383256-374d-49e2-bdf3-74a20ce00532","Type":"ContainerDied","Data":"1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97"}
Nov 02 08:09:49.584693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.584585    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6" event={"ID":"27383256-374d-49e2-bdf3-74a20ce00532","Type":"ContainerDied","Data":"b0880c08902bc4fc408bc3a5fa1f4230a7f4aea4671695eded03574755c02820"}
Nov 02 08:09:49.584693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.584615    4898 scope.go:122] "RemoveContainer" containerID="1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97"
Nov 02 08:09:49.596969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.596948    4898 scope.go:122] "RemoveContainer" containerID="1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97"
Nov 02 08:09:49.597313 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:09:49.597290    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97\": container with ID starting with 1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97 not found: ID does not exist" containerID="1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97"
Nov 02 08:09:49.597407 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.597318    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97"} err="failed to get container status \"1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97\": rpc error: code = NotFound desc = could not find container \"1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97\": container with ID starting with 1a8176f550a0786e165e9615e32b5c81d2e19ce55e46bef992258f3ec7e26e97 not found: ID does not exist"
Nov 02 08:09:49.645938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.645863    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"]
Nov 02 08:09:49.652990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:49.652954    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-696c7a/payload-processing-external-model-e2e-extmod-a-696c7a-67f84kwr6"]
Nov 02 08:09:50.507029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.506986    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"]
Nov 02 08:09:50.507746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.507726    4898 cpu_manager.go:414] "containerMap: removing container" podUID="de8f11aa-420b-4c67-aeb7-2eb5a62cebb5" containerName="revoke-keys"
Nov 02 08:09:50.507795 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.507747    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de8f11aa-420b-4c67-aeb7-2eb5a62cebb5" containerName="revoke-keys"
Nov 02 08:09:50.507795 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.507764    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7f28770f-1e0d-484e-b0ea-874058346bb6" containerName="payload-processing"
Nov 02 08:09:50.507795 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.507771    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f28770f-1e0d-484e-b0ea-874058346bb6" containerName="payload-processing"
Nov 02 08:09:50.507795 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.507786    4898 cpu_manager.go:414] "containerMap: removing container" podUID="27383256-374d-49e2-bdf3-74a20ce00532" containerName="payload-processing"
Nov 02 08:09:50.507795 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.507791    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27383256-374d-49e2-bdf3-74a20ce00532" containerName="payload-processing"
Nov 02 08:09:50.508000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.507799    4898 cpu_manager.go:414] "containerMap: removing container" podUID="741b8f34-3bac-4607-a2dc-e97073dd9582" containerName="revoke-keys"
Nov 02 08:09:50.508000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.507803    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741b8f34-3bac-4607-a2dc-e97073dd9582" containerName="revoke-keys"
Nov 02 08:09:50.508000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.507945    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="de8f11aa-420b-4c67-aeb7-2eb5a62cebb5" containerName="revoke-keys"
Nov 02 08:09:50.508000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.507954    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f28770f-1e0d-484e-b0ea-874058346bb6" containerName="payload-processing"
Nov 02 08:09:50.508000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.507964    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="27383256-374d-49e2-bdf3-74a20ce00532" containerName="payload-processing"
Nov 02 08:09:50.508000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.507969    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="741b8f34-3bac-4607-a2dc-e97073dd9582" containerName="revoke-keys"
Nov 02 08:09:50.513181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.513152    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.531819 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.531371    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 08:09:50.531819 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.531440    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 08:09:50.531819 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.531449    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 08:09:50.531819 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.531454    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 08:09:50.535985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.535954    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"]
Nov 02 08:09:50.583382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.583345    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-212653738a3c\""
Nov 02 08:09:50.583608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.583479    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-247a3d77fa68\""
Nov 02 08:09:50.583608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.583598    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-l2lh7\""
Nov 02 08:09:50.667664 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.667617    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-provider-credentials\") pod \"payload-processing-external-model-5b845f4bcb-qx9t6\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") " pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.667876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.667800    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-routing-overlay-volume\") pod \"payload-processing-external-model-5b845f4bcb-qx9t6\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") " pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.668047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.667954    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-plugins-config-volume\") pod \"payload-processing-external-model-5b845f4bcb-qx9t6\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") " pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.668149 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.668118    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzzk\" (UniqueName: \"kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-kube-api-access-vlzzk\") pod \"payload-processing-external-model-5b845f4bcb-qx9t6\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") " pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.769424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.769324    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzzk\" (UniqueName: \"kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-kube-api-access-vlzzk\") pod \"payload-processing-external-model-5b845f4bcb-qx9t6\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") " pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.769607 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.769489    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-provider-credentials\") pod \"payload-processing-external-model-5b845f4bcb-qx9t6\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") " pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.769702 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.769621    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-routing-overlay-volume\") pod \"payload-processing-external-model-5b845f4bcb-qx9t6\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") " pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.769702 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.769687    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-plugins-config-volume\") pod \"payload-processing-external-model-5b845f4bcb-qx9t6\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") " pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.770682 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.770644    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-routing-overlay-volume\") pod \"payload-processing-external-model-5b845f4bcb-qx9t6\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") " pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.770832 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.770754    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-plugins-config-volume\") pod \"payload-processing-external-model-5b845f4bcb-qx9t6\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") " pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.772626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.772596    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-provider-credentials\") pod \"payload-processing-external-model-5b845f4bcb-qx9t6\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") " pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.795413 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.795383    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzzk\" (UniqueName: \"kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-kube-api-access-vlzzk\") pod \"payload-processing-external-model-5b845f4bcb-qx9t6\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") " pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:50.829170 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:50.829138    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:51.192880 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:09:51.192840    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919e6da7_347b_4c84_adcf_8718daca938c.slice/crio-5ddbe2c6a462d6c2ad920c81d06c4b911bb58d44b3b09c65baba9f1f99d8ca91 WatchSource:0}: Error finding container 5ddbe2c6a462d6c2ad920c81d06c4b911bb58d44b3b09c65baba9f1f99d8ca91: Status 404 returned error can't find the container with id 5ddbe2c6a462d6c2ad920c81d06c4b911bb58d44b3b09c65baba9f1f99d8ca91
Nov 02 08:09:51.195954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:51.195927    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"]
Nov 02 08:09:51.470638 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:51.470550    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27383256-374d-49e2-bdf3-74a20ce00532" path="/var/lib/kubelet/pods/27383256-374d-49e2-bdf3-74a20ce00532/volumes"
Nov 02 08:09:51.622591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:51.622555    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6" event={"ID":"919e6da7-347b-4c84-adcf-8718daca938c","Type":"ContainerStarted","Data":"8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0"}
Nov 02 08:09:51.623071 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:51.622610    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6" event={"ID":"919e6da7-347b-4c84-adcf-8718daca938c","Type":"ContainerStarted","Data":"5ddbe2c6a462d6c2ad920c81d06c4b911bb58d44b3b09c65baba9f1f99d8ca91"}
Nov 02 08:09:51.623071 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:51.622714    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:09:51.671691 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:09:51.671630    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6" podStartSLOduration=1.67161339 podStartE2EDuration="1.67161339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:09:51.668288949 +0000 UTC m=+2742.358615914" watchObservedRunningTime="2026-11-02 08:09:51.67161339 +0000 UTC m=+2742.361940427"
Nov 02 08:10:02.040969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.040926    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"]
Nov 02 08:10:02.044573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.044539    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:10:02.101609 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.101564    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"]
Nov 02 08:10:02.108481 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.108449    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.132575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.132538    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-621795298e14\""
Nov 02 08:10:02.137913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.137506    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"]
Nov 02 08:10:02.223873 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.223831    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-provider-credentials\") pod \"payload-processing-external-model-5cf4bb9494-7n5wc\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.224110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.224009    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-routing-overlay-volume\") pod \"payload-processing-external-model-5cf4bb9494-7n5wc\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.224183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.224155    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-plugins-config-volume\") pod \"payload-processing-external-model-5cf4bb9494-7n5wc\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.224224 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.224208    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cclk\" (UniqueName: \"kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-kube-api-access-4cclk\") pod \"payload-processing-external-model-5cf4bb9494-7n5wc\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.325138 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.325046    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-routing-overlay-volume\") pod \"payload-processing-external-model-5cf4bb9494-7n5wc\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.325321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.325161    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-plugins-config-volume\") pod \"payload-processing-external-model-5cf4bb9494-7n5wc\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.325321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.325201    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cclk\" (UniqueName: \"kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-kube-api-access-4cclk\") pod \"payload-processing-external-model-5cf4bb9494-7n5wc\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.325453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.325385    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-provider-credentials\") pod \"payload-processing-external-model-5cf4bb9494-7n5wc\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.327591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.325808    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-routing-overlay-volume\") pod \"payload-processing-external-model-5cf4bb9494-7n5wc\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.327591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.326119    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-plugins-config-volume\") pod \"payload-processing-external-model-5cf4bb9494-7n5wc\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.329002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.328963    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-provider-credentials\") pod \"payload-processing-external-model-5cf4bb9494-7n5wc\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.350602 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.350560    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cclk\" (UniqueName: \"kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-kube-api-access-4cclk\") pod \"payload-processing-external-model-5cf4bb9494-7n5wc\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.430736 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.430697    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:02.768438 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.758499    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"]
Nov 02 08:10:02.774776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:02.774742    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc" event={"ID":"a9312130-d92b-490a-8781-013aa0a7a21c","Type":"ContainerStarted","Data":"7294b21ffea24a27e753cbef149c75c5da4894f23d9ebbb36c716d0e1bbab972"}
Nov 02 08:10:03.815736 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:03.815693    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc" event={"ID":"a9312130-d92b-490a-8781-013aa0a7a21c","Type":"ContainerStarted","Data":"8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f"}
Nov 02 08:10:03.817552 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:03.817526    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:05.871770 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:05.871700    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc" podStartSLOduration=3.871677032 podStartE2EDuration="3.871677032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:03.869321244 +0000 UTC m=+2754.559648212" watchObservedRunningTime="2026-11-02 08:10:05.871677032 +0000 UTC m=+2756.562003997"
Nov 02 08:10:05.873459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:05.873433    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"]
Nov 02 08:10:07.041512 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.041451    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6" podUID="919e6da7-347b-4c84-adcf-8718daca938c" containerName="payload-processing" containerID="cri-o://8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0" gracePeriod=25
Nov 02 08:10:07.339689 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.339654    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:10:07.438294 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.438250    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-kube-api-access-vlzzk\" (UniqueName: \"kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-kube-api-access-vlzzk\") pod \"919e6da7-347b-4c84-adcf-8718daca938c\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") "
Nov 02 08:10:07.438509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.438378    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-provider-credentials\") pod \"919e6da7-347b-4c84-adcf-8718daca938c\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") "
Nov 02 08:10:07.439952 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.438864    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-plugins-config-volume\") pod \"919e6da7-347b-4c84-adcf-8718daca938c\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") "
Nov 02 08:10:07.439952 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.438991    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-routing-overlay-volume\") pod \"919e6da7-347b-4c84-adcf-8718daca938c\" (UID: \"919e6da7-347b-4c84-adcf-8718daca938c\") "
Nov 02 08:10:07.440329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.440302    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-routing-overlay-volume" pod "919e6da7-347b-4c84-adcf-8718daca938c" (UID: "919e6da7-347b-4c84-adcf-8718daca938c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:10:07.440736 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.440693    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-plugins-config-volume" pod "919e6da7-347b-4c84-adcf-8718daca938c" (UID: "919e6da7-347b-4c84-adcf-8718daca938c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:10:07.441072 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.441043    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-provider-credentials" pod "919e6da7-347b-4c84-adcf-8718daca938c" (UID: "919e6da7-347b-4c84-adcf-8718daca938c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:10:07.441640 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.441611    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-kube-api-access-vlzzk" pod "919e6da7-347b-4c84-adcf-8718daca938c" (UID: "919e6da7-347b-4c84-adcf-8718daca938c"). InnerVolumeSpecName "kube-api-access-vlzzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:10:07.550985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.550085    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlzzk\" (UniqueName: \"kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-kube-api-access-vlzzk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:07.550985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.550130    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/919e6da7-347b-4c84-adcf-8718daca938c-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:07.550985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.550146    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:07.550985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.550161    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/919e6da7-347b-4c84-adcf-8718daca938c-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:07.854128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.854087    4898 generic.go:356] "Generic (PLEG): container finished" podID="919e6da7-347b-4c84-adcf-8718daca938c" containerID="8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0" exitCode=0
Nov 02 08:10:07.854331 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.854144    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"
Nov 02 08:10:07.854331 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.854170    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6" event={"ID":"919e6da7-347b-4c84-adcf-8718daca938c","Type":"ContainerDied","Data":"8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0"}
Nov 02 08:10:07.854331 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.854229    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6" event={"ID":"919e6da7-347b-4c84-adcf-8718daca938c","Type":"ContainerDied","Data":"5ddbe2c6a462d6c2ad920c81d06c4b911bb58d44b3b09c65baba9f1f99d8ca91"}
Nov 02 08:10:07.854331 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.854256    4898 scope.go:122] "RemoveContainer" containerID="8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0"
Nov 02 08:10:07.867982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.867948    4898 scope.go:122] "RemoveContainer" containerID="8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0"
Nov 02 08:10:07.868378 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:10:07.868352    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0\": container with ID starting with 8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0 not found: ID does not exist" containerID="8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0"
Nov 02 08:10:07.868498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.868386    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0"} err="failed to get container status \"8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0\": rpc error: code = NotFound desc = could not find container \"8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0\": container with ID starting with 8f58c42bdf8b166d2a96a56a1f17e70c94b96ff077ec67783f1917c86af4d3f0 not found: ID does not exist"
Nov 02 08:10:07.916055 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.916019    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"]
Nov 02 08:10:07.921173 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:07.921140    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b845f4bcb-qx9t6"]
Nov 02 08:10:08.356516 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.356474    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"]
Nov 02 08:10:08.357639 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.357613    4898 cpu_manager.go:414] "containerMap: removing container" podUID="919e6da7-347b-4c84-adcf-8718daca938c" containerName="payload-processing"
Nov 02 08:10:08.357735 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.357639    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919e6da7-347b-4c84-adcf-8718daca938c" containerName="payload-processing"
Nov 02 08:10:08.357840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.357827    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="919e6da7-347b-4c84-adcf-8718daca938c" containerName="payload-processing"
Nov 02 08:10:08.363716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.363688    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:08.407526 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.407483    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:10:08.407738 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.407622    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:10:08.408659 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.407490    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:10:08.408860 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.408834    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:10:08.412527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.412482    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"]
Nov 02 08:10:08.470631 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.470580    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c328dd-8075-4654-8b66-24920ae84817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec18d4e9-h57bb\" (UID: \"58c328dd-8075-4654-8b66-24920ae84817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:08.470841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.470814    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c328dd-8075-4654-8b66-24920ae84817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec18d4e9-h57bb\" (UID: \"58c328dd-8075-4654-8b66-24920ae84817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:08.471011 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.470984    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpt96\" (UniqueName: \"kubernetes.io/projected/58c328dd-8075-4654-8b66-24920ae84817-kube-api-access-fpt96\") pod \"maas-api-revoke-sub-ec18d4e9-h57bb\" (UID: \"58c328dd-8075-4654-8b66-24920ae84817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:08.572187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.572149    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpt96\" (UniqueName: \"kubernetes.io/projected/58c328dd-8075-4654-8b66-24920ae84817-kube-api-access-fpt96\") pod \"maas-api-revoke-sub-ec18d4e9-h57bb\" (UID: \"58c328dd-8075-4654-8b66-24920ae84817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:08.572408 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.572231    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c328dd-8075-4654-8b66-24920ae84817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec18d4e9-h57bb\" (UID: \"58c328dd-8075-4654-8b66-24920ae84817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:08.572602 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.572578    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c328dd-8075-4654-8b66-24920ae84817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec18d4e9-h57bb\" (UID: \"58c328dd-8075-4654-8b66-24920ae84817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:08.573368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.573341    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c328dd-8075-4654-8b66-24920ae84817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec18d4e9-h57bb\" (UID: \"58c328dd-8075-4654-8b66-24920ae84817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:08.575550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.575521    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c328dd-8075-4654-8b66-24920ae84817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec18d4e9-h57bb\" (UID: \"58c328dd-8075-4654-8b66-24920ae84817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:08.596536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.596493    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpt96\" (UniqueName: \"kubernetes.io/projected/58c328dd-8075-4654-8b66-24920ae84817-kube-api-access-fpt96\") pod \"maas-api-revoke-sub-ec18d4e9-h57bb\" (UID: \"58c328dd-8075-4654-8b66-24920ae84817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:08.709323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:08.709286    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:09.023432 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:10:09.023398    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c328dd_8075_4654_8b66_24920ae84817.slice/crio-ea966ab25774274c20f4ac2a1c88fe6e9048e0acc48fed71876c072249645bd8 WatchSource:0}: Error finding container ea966ab25774274c20f4ac2a1c88fe6e9048e0acc48fed71876c072249645bd8: Status 404 returned error can't find the container with id ea966ab25774274c20f4ac2a1c88fe6e9048e0acc48fed71876c072249645bd8
Nov 02 08:10:09.024866 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:09.024842    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"]
Nov 02 08:10:09.469957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:09.469929    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919e6da7-347b-4c84-adcf-8718daca938c" path="/var/lib/kubelet/pods/919e6da7-347b-4c84-adcf-8718daca938c/volumes"
Nov 02 08:10:09.909224 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:09.909174    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb" event={"ID":"58c328dd-8075-4654-8b66-24920ae84817","Type":"ContainerStarted","Data":"79b30f3d15f5a290e2bb13e243ab86b8efc7de5577278994a5c94d844ac67168"}
Nov 02 08:10:09.909444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:09.909254    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb" event={"ID":"58c328dd-8075-4654-8b66-24920ae84817","Type":"ContainerStarted","Data":"ea966ab25774274c20f4ac2a1c88fe6e9048e0acc48fed71876c072249645bd8"}
Nov 02 08:10:09.958028 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:09.957981    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb" podStartSLOduration=1.957967183 podStartE2EDuration="1.957967183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:09.951618115 +0000 UTC m=+2760.641945098" watchObservedRunningTime="2026-11-02 08:10:09.957967183 +0000 UTC m=+2760.648294146"
Nov 02 08:10:10.920114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:10.920074    4898 generic.go:356] "Generic (PLEG): container finished" podID="58c328dd-8075-4654-8b66-24920ae84817" containerID="79b30f3d15f5a290e2bb13e243ab86b8efc7de5577278994a5c94d844ac67168" exitCode=0
Nov 02 08:10:10.920554 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:10.920165    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb" event={"ID":"58c328dd-8075-4654-8b66-24920ae84817","Type":"ContainerDied","Data":"79b30f3d15f5a290e2bb13e243ab86b8efc7de5577278994a5c94d844ac67168"}
Nov 02 08:10:11.844979 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:11.844929    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc" podUID="a9312130-d92b-490a-8781-013aa0a7a21c" containerName="payload-processing" containerID="cri-o://8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f" gracePeriod=25
Nov 02 08:10:12.116011 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.115921    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"]
Nov 02 08:10:12.122130 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.122103    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.139797 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.139759    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"]
Nov 02 08:10:12.219206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.219137    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:12.224044 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.224019    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:12.270429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.270379    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-provider-credentials\") pod \"payload-processing-external-model-5cf4bb9494-z68c2\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.270606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.270578    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-routing-overlay-volume\") pod \"payload-processing-external-model-5cf4bb9494-z68c2\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.270757 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.270614    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhw44\" (UniqueName: \"kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-kube-api-access-qhw44\") pod \"payload-processing-external-model-5cf4bb9494-z68c2\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.270757 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.270701    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-plugins-config-volume\") pod \"payload-processing-external-model-5cf4bb9494-z68c2\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.371533 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.371434    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58c328dd-8075-4654-8b66-24920ae84817-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c328dd-8075-4654-8b66-24920ae84817-maas-api-xks-ca\") pod \"58c328dd-8075-4654-8b66-24920ae84817\" (UID: \"58c328dd-8075-4654-8b66-24920ae84817\") "
Nov 02 08:10:12.371719 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.371593    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-routing-overlay-volume\") pod \"a9312130-d92b-490a-8781-013aa0a7a21c\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") "
Nov 02 08:10:12.371719 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.371651    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-provider-credentials\") pod \"a9312130-d92b-490a-8781-013aa0a7a21c\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") "
Nov 02 08:10:12.371719 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.371684    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-plugins-config-volume\") pod \"a9312130-d92b-490a-8781-013aa0a7a21c\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") "
Nov 02 08:10:12.372032 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.371804    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-kube-api-access-4cclk\" (UniqueName: \"kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-kube-api-access-4cclk\") pod \"a9312130-d92b-490a-8781-013aa0a7a21c\" (UID: \"a9312130-d92b-490a-8781-013aa0a7a21c\") "
Nov 02 08:10:12.372032 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.371842    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c328dd-8075-4654-8b66-24920ae84817-kube-api-access-fpt96\" (UniqueName: \"kubernetes.io/projected/58c328dd-8075-4654-8b66-24920ae84817-kube-api-access-fpt96\") pod \"58c328dd-8075-4654-8b66-24920ae84817\" (UID: \"58c328dd-8075-4654-8b66-24920ae84817\") "
Nov 02 08:10:12.372179 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.372109    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c328dd-8075-4654-8b66-24920ae84817-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c328dd-8075-4654-8b66-24920ae84817-maas-api-openshift-ca\") pod \"58c328dd-8075-4654-8b66-24920ae84817\" (UID: \"58c328dd-8075-4654-8b66-24920ae84817\") "
Nov 02 08:10:12.372594 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.372564    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-provider-credentials\") pod \"payload-processing-external-model-5cf4bb9494-z68c2\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.372684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.372665    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-routing-overlay-volume\") pod \"payload-processing-external-model-5cf4bb9494-z68c2\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.372746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.372697    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhw44\" (UniqueName: \"kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-kube-api-access-qhw44\") pod \"payload-processing-external-model-5cf4bb9494-z68c2\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.372811 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.372776    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-plugins-config-volume\") pod \"payload-processing-external-model-5cf4bb9494-z68c2\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.373688 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.373662    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-plugins-config-volume\") pod \"payload-processing-external-model-5cf4bb9494-z68c2\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.374011 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.373984    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-routing-overlay-volume" pod "a9312130-d92b-490a-8781-013aa0a7a21c" (UID: "a9312130-d92b-490a-8781-013aa0a7a21c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:10:12.374426 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.374325    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c328dd-8075-4654-8b66-24920ae84817-maas-api-xks-ca" pod "58c328dd-8075-4654-8b66-24920ae84817" (UID: "58c328dd-8075-4654-8b66-24920ae84817"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:10:12.374685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.374630    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c328dd-8075-4654-8b66-24920ae84817-maas-api-openshift-ca" pod "58c328dd-8075-4654-8b66-24920ae84817" (UID: "58c328dd-8075-4654-8b66-24920ae84817"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:10:12.374775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.374680    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-plugins-config-volume" pod "a9312130-d92b-490a-8781-013aa0a7a21c" (UID: "a9312130-d92b-490a-8781-013aa0a7a21c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:10:12.375391 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.375363    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-routing-overlay-volume\") pod \"payload-processing-external-model-5cf4bb9494-z68c2\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.380351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.380313    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-kube-api-access-4cclk" pod "a9312130-d92b-490a-8781-013aa0a7a21c" (UID: "a9312130-d92b-490a-8781-013aa0a7a21c"). InnerVolumeSpecName "kube-api-access-4cclk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:10:12.380458 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.380403    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c328dd-8075-4654-8b66-24920ae84817-kube-api-access-fpt96" pod "58c328dd-8075-4654-8b66-24920ae84817" (UID: "58c328dd-8075-4654-8b66-24920ae84817"). InnerVolumeSpecName "kube-api-access-fpt96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:10:12.380529 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.380460    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-provider-credentials" pod "a9312130-d92b-490a-8781-013aa0a7a21c" (UID: "a9312130-d92b-490a-8781-013aa0a7a21c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:10:12.382488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.382460    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-provider-credentials\") pod \"payload-processing-external-model-5cf4bb9494-z68c2\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.399460 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.399431    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhw44\" (UniqueName: \"kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-kube-api-access-qhw44\") pod \"payload-processing-external-model-5cf4bb9494-z68c2\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") " pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.474209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.474170    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cclk\" (UniqueName: \"kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-kube-api-access-4cclk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:12.474209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.474201    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpt96\" (UniqueName: \"kubernetes.io/projected/58c328dd-8075-4654-8b66-24920ae84817-kube-api-access-fpt96\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:12.474209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.474210    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c328dd-8075-4654-8b66-24920ae84817-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:12.474453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.474221    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c328dd-8075-4654-8b66-24920ae84817-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:12.474453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.474230    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:12.474453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.474238    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9312130-d92b-490a-8781-013aa0a7a21c-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:12.474453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.474248    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9312130-d92b-490a-8781-013aa0a7a21c-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:12.515871 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.515837    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.829601 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:10:12.829558    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90092965_2956_402f_bc2a_40455bfe795b.slice/crio-370c203b88bcd5feb2c85395265f9f1d078669e9f1b6a0f28930fcf55016e06b WatchSource:0}: Error finding container 370c203b88bcd5feb2c85395265f9f1d078669e9f1b6a0f28930fcf55016e06b: Status 404 returned error can't find the container with id 370c203b88bcd5feb2c85395265f9f1d078669e9f1b6a0f28930fcf55016e06b
Nov 02 08:10:12.832434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.832409    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"]
Nov 02 08:10:12.884999 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.884459    4898 scope.go:122] "RemoveContainer" containerID="942f94cb2a2a573637976c3695e4d66fc3d79373d6390ba2b57a8a3ffef88251"
Nov 02 08:10:12.974979 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.974945    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2" event={"ID":"90092965-2956-402f-bc2a-40455bfe795b","Type":"ContainerStarted","Data":"3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7"}
Nov 02 08:10:12.975119 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.975020    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:12.975119 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.975048    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2" event={"ID":"90092965-2956-402f-bc2a-40455bfe795b","Type":"ContainerStarted","Data":"370c203b88bcd5feb2c85395265f9f1d078669e9f1b6a0f28930fcf55016e06b"}
Nov 02 08:10:12.976546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.976516    4898 generic.go:356] "Generic (PLEG): container finished" podID="a9312130-d92b-490a-8781-013aa0a7a21c" containerID="8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f" exitCode=0
Nov 02 08:10:12.976680 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.976557    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"
Nov 02 08:10:12.976680 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.976590    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc" event={"ID":"a9312130-d92b-490a-8781-013aa0a7a21c","Type":"ContainerDied","Data":"8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f"}
Nov 02 08:10:12.976680 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.976660    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc" event={"ID":"a9312130-d92b-490a-8781-013aa0a7a21c","Type":"ContainerDied","Data":"7294b21ffea24a27e753cbef149c75c5da4894f23d9ebbb36c716d0e1bbab972"}
Nov 02 08:10:12.976867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.976691    4898 scope.go:122] "RemoveContainer" containerID="8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f"
Nov 02 08:10:12.978546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.978521    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb" event={"ID":"58c328dd-8075-4654-8b66-24920ae84817","Type":"ContainerDied","Data":"ea966ab25774274c20f4ac2a1c88fe6e9048e0acc48fed71876c072249645bd8"}
Nov 02 08:10:12.978660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.978566    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea966ab25774274c20f4ac2a1c88fe6e9048e0acc48fed71876c072249645bd8"
Nov 02 08:10:12.978660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.978624    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"
Nov 02 08:10:12.989879 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.989859    4898 scope.go:122] "RemoveContainer" containerID="8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f"
Nov 02 08:10:12.990569 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:10:12.990536    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f\": container with ID starting with 8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f not found: ID does not exist" containerID="8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f"
Nov 02 08:10:12.990645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:12.990573    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f"} err="failed to get container status \"8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f\": rpc error: code = NotFound desc = could not find container \"8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f\": container with ID starting with 8c09b02e29d1c96b63134b50745135062e2bb77a393d153d543d2eac6c2b8b9f not found: ID does not exist"
Nov 02 08:10:13.020820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:13.020766    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2" podStartSLOduration=1.020751267 podStartE2EDuration="1.020751267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:13.019138907 +0000 UTC m=+2763.709465872" watchObservedRunningTime="2026-11-02 08:10:13.020751267 +0000 UTC m=+2763.711078231"
Nov 02 08:10:13.065517 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:13.065478    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"]
Nov 02 08:10:13.072844 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:13.072804    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cf4bb9494-7n5wc"]
Nov 02 08:10:13.471350 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:13.471308    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9312130-d92b-490a-8781-013aa0a7a21c" path="/var/lib/kubelet/pods/a9312130-d92b-490a-8781-013aa0a7a21c/volumes"
Nov 02 08:10:15.044993 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:10:15.044948    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:10:16.006413 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.006292    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"]
Nov 02 08:10:16.007225 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.007202    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a9312130-d92b-490a-8781-013aa0a7a21c" containerName="payload-processing"
Nov 02 08:10:16.007225 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.007225    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9312130-d92b-490a-8781-013aa0a7a21c" containerName="payload-processing"
Nov 02 08:10:16.007440 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.007308    4898 cpu_manager.go:414] "containerMap: removing container" podUID="58c328dd-8075-4654-8b66-24920ae84817" containerName="revoke-keys"
Nov 02 08:10:16.007440 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.007317    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c328dd-8075-4654-8b66-24920ae84817" containerName="revoke-keys"
Nov 02 08:10:16.007551 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.007496    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c328dd-8075-4654-8b66-24920ae84817" containerName="revoke-keys"
Nov 02 08:10:16.007551 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.007519    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9312130-d92b-490a-8781-013aa0a7a21c" containerName="payload-processing"
Nov 02 08:10:16.013253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.013231    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.041916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.035244    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"]
Nov 02 08:10:16.045025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.042279    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"openshift-service-ca.crt\""
Nov 02 08:10:16.045025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.043018    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"kube-root-ca.crt\""
Nov 02 08:10:16.045025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.043187    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"routing-overlay\""
Nov 02 08:10:16.045025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.043255    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0f2e06\""
Nov 02 08:10:16.072102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.072065    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"agc-cred-e2eab36a85e6\""
Nov 02 08:10:16.072343 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.072215    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"payload-processing-external-model-e2e--9b11fa18-dockercfg-244tn\""
Nov 02 08:10:16.074674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.074634    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.074828 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.074787    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.074828 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.074817    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.075032 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.075005    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx8hc\" (UniqueName: \"kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-kube-api-access-kx8hc\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.190911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.178792    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx8hc\" (UniqueName: \"kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-kube-api-access-kx8hc\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.190911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.178981    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.190911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.179146    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.190911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.179180    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.190911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.188342    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.190911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.189115    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.225921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.218127    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.256843 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.239030    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx8hc\" (UniqueName: \"kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-kube-api-access-kx8hc\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.348791 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.348756    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:16.731418 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:10:16.731382    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d117963_b74e_4be4_885e_56cb6a205310.slice/crio-e563188363454ba8dbf84499172fdabdfd3c2ec3df58171781f422eef9f15a9a WatchSource:0}: Error finding container e563188363454ba8dbf84499172fdabdfd3c2ec3df58171781f422eef9f15a9a: Status 404 returned error can't find the container with id e563188363454ba8dbf84499172fdabdfd3c2ec3df58171781f422eef9f15a9a
Nov 02 08:10:16.734358 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:16.734286    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"]
Nov 02 08:10:17.081416 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:17.081328    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl" event={"ID":"3d117963-b74e-4be4-885e-56cb6a205310","Type":"ContainerStarted","Data":"b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7"}
Nov 02 08:10:17.081416 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:17.081393    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:17.081416 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:17.081410    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl" event={"ID":"3d117963-b74e-4be4-885e-56cb6a205310","Type":"ContainerStarted","Data":"e563188363454ba8dbf84499172fdabdfd3c2ec3df58171781f422eef9f15a9a"}
Nov 02 08:10:17.132360 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:17.132309    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl" podStartSLOduration=2.132296261 podStartE2EDuration="2.132296261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:17.130879232 +0000 UTC m=+2767.821206196" watchObservedRunningTime="2026-11-02 08:10:17.132296261 +0000 UTC m=+2767.822623225"
Nov 02 08:10:22.212915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:22.212854    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"]
Nov 02 08:10:22.216952 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:22.216926    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:23.773881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:23.773843    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"]
Nov 02 08:10:23.779398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:23.779348    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:23.833245 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:23.825173    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"]
Nov 02 08:10:23.897982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:23.897944    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717e0316-0675-4f60-ba3e-e51e854cfc72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:23.898163 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:23.898002    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:23.898163 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:23.898119    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vc9\" (UniqueName: \"kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-kube-api-access-24vc9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:23.898254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:23.898165    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:23.898293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:23.898252    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:23.898408 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:23.898392    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:23.898470 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:23.898451    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:23.898542 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:23.898516    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:23.898591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:23.898557    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.000234 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.000170    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.000449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.000261    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vc9\" (UniqueName: \"kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-kube-api-access-24vc9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.000449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.000295    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.000449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.000320    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.000449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.000439    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.000687 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.000467    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.000687 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.000562    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.000687 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.000610    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.000840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.000759    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.000911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.000840    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717e0316-0675-4f60-ba3e-e51e854cfc72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.001013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.000994    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.001115 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.001089    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.001186 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.001150    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.001584 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.001561    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717e0316-0675-4f60-ba3e-e51e854cfc72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.002972 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.002943    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.003069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.002981    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.026616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.026547    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.026828 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.026808    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vc9\" (UniqueName: \"kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-kube-api-access-24vc9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jvnf9\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.109670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.106756    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:24.441945 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:10:24.441914    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717e0316_0675_4f60_ba3e_e51e854cfc72.slice/crio-6052f4ca9f4e6016821cd78c694ff43dc8555223995d404a9b9c55902e83dba6 WatchSource:0}: Error finding container 6052f4ca9f4e6016821cd78c694ff43dc8555223995d404a9b9c55902e83dba6: Status 404 returned error can't find the container with id 6052f4ca9f4e6016821cd78c694ff43dc8555223995d404a9b9c55902e83dba6
Nov 02 08:10:24.445005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.444965    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:10:24.445083 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.445042    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:10:24.445083 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.445072    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:10:24.446630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:24.446606    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"]
Nov 02 08:10:25.260418 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:25.260376    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" event={"ID":"717e0316-0675-4f60-ba3e-e51e854cfc72","Type":"ContainerStarted","Data":"e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602"}
Nov 02 08:10:25.260835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:25.260444    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" event={"ID":"717e0316-0675-4f60-ba3e-e51e854cfc72","Type":"ContainerStarted","Data":"6052f4ca9f4e6016821cd78c694ff43dc8555223995d404a9b9c55902e83dba6"}
Nov 02 08:10:25.332683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:25.332620    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" podStartSLOduration=2.332605466 podStartE2EDuration="2.332605466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:25.327270819 +0000 UTC m=+2776.017597796" watchObservedRunningTime="2026-11-02 08:10:25.332605466 +0000 UTC m=+2776.022932429"
Nov 02 08:10:26.107910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:26.107285    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:26.118916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:26.117196    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jvnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body=
Nov 02 08:10:26.118916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:26.117259    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" podUID="717e0316-0675-4f60-ba3e-e51e854cfc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused"
Nov 02 08:10:27.108055 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.108020    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jvnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body=
Nov 02 08:10:27.108477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.108073    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" podUID="717e0316-0675-4f60-ba3e-e51e854cfc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused"
Nov 02 08:10:27.213564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.213505    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2" podUID="90092965-2956-402f-bc2a-40455bfe795b" containerName="payload-processing" containerID="cri-o://3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7" gracePeriod=25
Nov 02 08:10:27.688961 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.685176    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"]
Nov 02 08:10:27.699704 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.699679    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:27.702261 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.702232    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"]
Nov 02 08:10:27.710301 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.710268    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:27.710442 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.710398    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-958q4\" (UniqueName: \"kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-kube-api-access-958q4\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:27.710565 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.710549    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:27.710654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.710582    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:27.716933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.716147    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"agc-cred-f45c1914cd99\""
Nov 02 08:10:27.728381 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.728354    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-e2e--71cd8499-dockercfg-hftbt\""
Nov 02 08:10:27.729078 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.729051    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"openshift-service-ca.crt\""
Nov 02 08:10:27.729180 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.729135    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-736f7c\""
Nov 02 08:10:27.729256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.729199    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"routing-overlay\""
Nov 02 08:10:27.729319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.729285    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"kube-root-ca.crt\""
Nov 02 08:10:27.890911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.868004    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:27.890911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.868158    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-958q4\" (UniqueName: \"kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-kube-api-access-958q4\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:27.890911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.868280    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:27.890911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.868305    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:27.890911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.869269    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:27.890911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.873378    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:27.894721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.894338    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:27.939964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:27.939606    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:28.021958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.021651    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-958q4\" (UniqueName: \"kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-kube-api-access-958q4\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:28.094731 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.094693    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:10:28.107723 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.107695    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jvnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body=
Nov 02 08:10:28.107869 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.107746    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" podUID="717e0316-0675-4f60-ba3e-e51e854cfc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused"
Nov 02 08:10:28.121214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.121182    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-provider-credentials\") pod \"90092965-2956-402f-bc2a-40455bfe795b\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") "
Nov 02 08:10:28.121655 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.121372    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-plugins-config-volume\") pod \"90092965-2956-402f-bc2a-40455bfe795b\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") "
Nov 02 08:10:28.121655 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.121416    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-routing-overlay-volume\") pod \"90092965-2956-402f-bc2a-40455bfe795b\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") "
Nov 02 08:10:28.121655 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.121622    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-kube-api-access-qhw44\" (UniqueName: \"kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-kube-api-access-qhw44\") pod \"90092965-2956-402f-bc2a-40455bfe795b\" (UID: \"90092965-2956-402f-bc2a-40455bfe795b\") "
Nov 02 08:10:28.121875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.121846    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-routing-overlay-volume" pod "90092965-2956-402f-bc2a-40455bfe795b" (UID: "90092965-2956-402f-bc2a-40455bfe795b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:10:28.122172 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.122145    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-plugins-config-volume" pod "90092965-2956-402f-bc2a-40455bfe795b" (UID: "90092965-2956-402f-bc2a-40455bfe795b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:10:28.122454 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.122429    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:28.122531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.122463    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90092965-2956-402f-bc2a-40455bfe795b-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:28.123766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.123739    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-provider-credentials" pod "90092965-2956-402f-bc2a-40455bfe795b" (UID: "90092965-2956-402f-bc2a-40455bfe795b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:10:28.123872 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.123768    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-kube-api-access-qhw44" pod "90092965-2956-402f-bc2a-40455bfe795b" (UID: "90092965-2956-402f-bc2a-40455bfe795b"). InnerVolumeSpecName "kube-api-access-qhw44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:10:28.192470 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.192365    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:28.224288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.224259    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhw44\" (UniqueName: \"kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-kube-api-access-qhw44\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:28.224288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.224290    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90092965-2956-402f-bc2a-40455bfe795b-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:10:28.319238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.318356    4898 generic.go:356] "Generic (PLEG): container finished" podID="90092965-2956-402f-bc2a-40455bfe795b" containerID="3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7" exitCode=0
Nov 02 08:10:28.319238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.318472    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"
Nov 02 08:10:28.319238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.318668    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2" event={"ID":"90092965-2956-402f-bc2a-40455bfe795b","Type":"ContainerDied","Data":"3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7"}
Nov 02 08:10:28.319238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.318725    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2" event={"ID":"90092965-2956-402f-bc2a-40455bfe795b","Type":"ContainerDied","Data":"370c203b88bcd5feb2c85395265f9f1d078669e9f1b6a0f28930fcf55016e06b"}
Nov 02 08:10:28.319238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.318752    4898 scope.go:122] "RemoveContainer" containerID="3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7"
Nov 02 08:10:28.334356 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.334042    4898 scope.go:122] "RemoveContainer" containerID="3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7"
Nov 02 08:10:28.334526 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:10:28.334501    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7\": container with ID starting with 3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7 not found: ID does not exist" containerID="3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7"
Nov 02 08:10:28.334602 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.334542    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7"} err="failed to get container status \"3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7\": rpc error: code = NotFound desc = could not find container \"3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7\": container with ID starting with 3f6efc8a1c779d0e7925c31f5ab9f22b6ac169db4e5a6b4ba9a7b35058e76ca7 not found: ID does not exist"
Nov 02 08:10:28.396851 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.395922    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"]
Nov 02 08:10:28.403509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.403235    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cf4bb9494-z68c2"]
Nov 02 08:10:28.552031 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:10:28.551984    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a6eabb_e243_437a_9eb2_f048ea5ab124.slice/crio-d2c0e0a2e4f7a47695dd1204a8b4e660025ffa227476b370d0f42b95913a1db4 WatchSource:0}: Error finding container d2c0e0a2e4f7a47695dd1204a8b4e660025ffa227476b370d0f42b95913a1db4: Status 404 returned error can't find the container with id d2c0e0a2e4f7a47695dd1204a8b4e660025ffa227476b370d0f42b95913a1db4
Nov 02 08:10:28.576938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.555715    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"]
Nov 02 08:10:28.831978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.831692    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"]
Nov 02 08:10:28.834958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.832824    4898 cpu_manager.go:414] "containerMap: removing container" podUID="90092965-2956-402f-bc2a-40455bfe795b" containerName="payload-processing"
Nov 02 08:10:28.834958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.832850    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90092965-2956-402f-bc2a-40455bfe795b" containerName="payload-processing"
Nov 02 08:10:28.834958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.833235    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="90092965-2956-402f-bc2a-40455bfe795b" containerName="payload-processing"
Nov 02 08:10:28.848789 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.848727    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"]
Nov 02 08:10:28.848965 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.848858    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:28.880489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.880464    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"openshift-service-ca.crt\""
Nov 02 08:10:28.883779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.880801    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-e2e--1a3816b7-dockercfg-52g6v\""
Nov 02 08:10:28.887090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.882965    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"agc-cred-a89b1ea64886\""
Nov 02 08:10:28.887090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.883075    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"routing-overlay\""
Nov 02 08:10:28.887090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.883132    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2dde84\""
Nov 02 08:10:28.887090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:28.883207    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"kube-root-ca.crt\""
Nov 02 08:10:29.040914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.040694    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.041110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.041053    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqs82\" (UniqueName: \"kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-kube-api-access-rqs82\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.041110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.041100    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.041258 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.041152    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.109975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.109374    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jvnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body=
Nov 02 08:10:29.109975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.109451    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" podUID="717e0316-0675-4f60-ba3e-e51e854cfc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused"
Nov 02 08:10:29.153905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.144819    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.153905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.144934    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.153905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.145020    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.153905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.145296    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqs82\" (UniqueName: \"kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-kube-api-access-rqs82\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.153905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.146449    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.153905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.147696    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.153905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.150431    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.182839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.182802    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqs82\" (UniqueName: \"kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-kube-api-access-rqs82\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.200290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.198233    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:29.332877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.332844    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn" event={"ID":"49a6eabb-e243-437a-9eb2-f048ea5ab124","Type":"ContainerStarted","Data":"4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75"}
Nov 02 08:10:29.333074 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.332914    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn" event={"ID":"49a6eabb-e243-437a-9eb2-f048ea5ab124","Type":"ContainerStarted","Data":"d2c0e0a2e4f7a47695dd1204a8b4e660025ffa227476b370d0f42b95913a1db4"}
Nov 02 08:10:29.333814 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.333795    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:29.392738 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.392615    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn" podStartSLOduration=2.392597197 podStartE2EDuration="2.392597197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:29.386446404 +0000 UTC m=+2780.076773369" watchObservedRunningTime="2026-11-02 08:10:29.392597197 +0000 UTC m=+2780.082924162"
Nov 02 08:10:29.525912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.521446    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90092965-2956-402f-bc2a-40455bfe795b" path="/var/lib/kubelet/pods/90092965-2956-402f-bc2a-40455bfe795b/volumes"
Nov 02 08:10:29.565285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:29.562110    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"]
Nov 02 08:10:29.588526 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:10:29.588471    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f4a436_b299_4503_9c1f_cd8ef0baff9e.slice/crio-5c123c285110f95bcf06d0d8727c92d3e9f1d11086d22e93abe45af21ce3d280 WatchSource:0}: Error finding container 5c123c285110f95bcf06d0d8727c92d3e9f1d11086d22e93abe45af21ce3d280: Status 404 returned error can't find the container with id 5c123c285110f95bcf06d0d8727c92d3e9f1d11086d22e93abe45af21ce3d280
Nov 02 08:10:30.107637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:30.107528    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jvnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body=
Nov 02 08:10:30.107637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:30.107595    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" podUID="717e0316-0675-4f60-ba3e-e51e854cfc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused"
Nov 02 08:10:30.348456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:30.347688    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9" event={"ID":"e6f4a436-b299-4503-9c1f-cd8ef0baff9e","Type":"ContainerStarted","Data":"ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869"}
Nov 02 08:10:30.348456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:30.347874    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:30.348456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:30.347959    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9" event={"ID":"e6f4a436-b299-4503-9c1f-cd8ef0baff9e","Type":"ContainerStarted","Data":"5c123c285110f95bcf06d0d8727c92d3e9f1d11086d22e93abe45af21ce3d280"}
Nov 02 08:10:30.402437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:30.402384    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9" podStartSLOduration=2.4023681310000002 podStartE2EDuration="2.402368131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:30.39756128 +0000 UTC m=+2781.087888264" watchObservedRunningTime="2026-11-02 08:10:30.402368131 +0000 UTC m=+2781.092695127"
Nov 02 08:10:31.107271 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:31.107230    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jvnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body=
Nov 02 08:10:31.107467 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:31.107294    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" podUID="717e0316-0675-4f60-ba3e-e51e854cfc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused"
Nov 02 08:10:32.107661 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:32.107627    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jvnf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused" start-of-body=
Nov 02 08:10:32.108088 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:32.107680    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" podUID="717e0316-0675-4f60-ba3e-e51e854cfc72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.55:15021/healthz/ready\": dial tcp 10.132.1.55:15021: connect: connection refused"
Nov 02 08:10:33.527194 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:33.527130    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:33.529421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:33.529394    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:33.530969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:33.530950    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:10:34.070422 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:34.070380    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"]
Nov 02 08:10:34.079114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:34.079077    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77714a05-8jncb"]
Nov 02 08:10:35.467441 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:35.467401    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95d37ea-b122-4c23-beed-9e888217d3bf" path="/var/lib/kubelet/pods/e95d37ea-b122-4c23-beed-9e888217d3bf/volumes"
Nov 02 08:10:38.836518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:38.836482    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"]
Nov 02 08:10:38.843379 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:38.843355    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:38.877501 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:38.877470    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"]
Nov 02 08:10:38.932976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:38.932931    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65nln\" (UniqueName: \"kubernetes.io/projected/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-kube-api-access-65nln\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:38.933150 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:38.932994    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:38.933150 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:38.933100    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:38.933299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:38.933267    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:38.933429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:38.933409    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:38.933502 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:38.933487    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:38.933614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:38.933594    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:38.933670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:38.933653    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:38.933733 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:38.933704    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.034820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.034772    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65nln\" (UniqueName: \"kubernetes.io/projected/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-kube-api-access-65nln\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.034820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.034825    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.035104 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.034863    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.035104 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.035019    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.035188 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.035118    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.035188 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.035173    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.035284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.035244    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.035339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.035292    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.035339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.035324    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.035455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.035344    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.035455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.035366    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.035580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.035459    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.035643 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.035616    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.035970 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.035943    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.037726 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.037703    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.037829 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.037810    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.069206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.069168    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.073236 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.073210    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65nln\" (UniqueName: \"kubernetes.io/projected/a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb-kube-api-access-65nln\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m4rk6\" (UID: \"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.158237 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.158205    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:39.501494 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:10:39.501458    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f8714b_4e0a_4515_bf6a_6d53f8b2cdcb.slice/crio-ac84a48b8ab39c3620783da1e7d0d89c5cdc81a2ff30826d276b0281f4e43dde WatchSource:0}: Error finding container ac84a48b8ab39c3620783da1e7d0d89c5cdc81a2ff30826d276b0281f4e43dde: Status 404 returned error can't find the container with id ac84a48b8ab39c3620783da1e7d0d89c5cdc81a2ff30826d276b0281f4e43dde
Nov 02 08:10:39.504176 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.504150    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"]
Nov 02 08:10:39.505619 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.505577    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:10:39.505721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.505664    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:10:39.505721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:39.505710    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:10:40.447434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:40.447389    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" event={"ID":"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb","Type":"ContainerStarted","Data":"c592e9644e52a2cb0f954e99c69fcbd7d61c107538b74444977e7955d0ee8c40"}
Nov 02 08:10:40.447858 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:40.447452    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" event={"ID":"a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb","Type":"ContainerStarted","Data":"ac84a48b8ab39c3620783da1e7d0d89c5cdc81a2ff30826d276b0281f4e43dde"}
Nov 02 08:10:40.528494 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:40.528439    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podStartSLOduration=2.528423354 podStartE2EDuration="2.528423354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:10:40.525242448 +0000 UTC m=+2791.215569432" watchObservedRunningTime="2026-11-02 08:10:40.528423354 +0000 UTC m=+2791.218750318"
Nov 02 08:10:41.160797 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:41.159260    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:41.182915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:41.168005    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Nov 02 08:10:41.182915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:41.168084    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Nov 02 08:10:41.379161 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:41.379124    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:10:41.383108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:41.383084    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:10:42.163514 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:42.163455    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Nov 02 08:10:42.164276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:42.164240    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Nov 02 08:10:43.158734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:43.158682    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Nov 02 08:10:43.159064 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:43.159019    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Nov 02 08:10:44.160333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:44.160292    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Nov 02 08:10:44.160978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:44.160941    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Nov 02 08:10:45.048174 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:45.048140    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"]
Nov 02 08:10:45.054394 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:45.054366    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e8be127-wtnmw"]
Nov 02 08:10:45.111915 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:10:45.095105    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:10:45.158865 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:45.158818    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Nov 02 08:10:45.159094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:45.158906    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Nov 02 08:10:45.538921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:45.538206    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78356009-a9e0-404d-a82b-0518bfa85ba3" path="/var/lib/kubelet/pods/78356009-a9e0-404d-a82b-0518bfa85ba3/volumes"
Nov 02 08:10:46.163918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:46.162320    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Nov 02 08:10:46.163918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:46.162388    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Nov 02 08:10:47.159169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:47.159127    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Nov 02 08:10:47.159686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:47.159188    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Nov 02 08:10:48.158898 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:48.158852    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused" start-of-body=
Nov 02 08:10:48.159093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:48.158962    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": dial tcp 10.132.1.58:15021: connect: connection refused"
Nov 02 08:10:50.170039 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:50.169993    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:10:50.170611 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:50.170575    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:10:51.213033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:51.212976    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:10:51.213553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:51.213063    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:10:52.074403 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:52.074357    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:52.075047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:52.075022    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:52.076023 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:52.075998    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6"
Nov 02 08:10:58.044106 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:58.044072    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"]
Nov 02 08:10:58.048973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:58.048943    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-576a953c-j27kh"]
Nov 02 08:10:59.467506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:10:59.467470    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f526e30c-847e-4602-b4b1-f272c4e46c96" path="/var/lib/kubelet/pods/f526e30c-847e-4602-b4b1-f272c4e46c96/volumes"
Nov 02 08:11:05.033069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:05.033032    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"]
Nov 02 08:11:05.044797 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:05.044768    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b0a3b4-db4zz"]
Nov 02 08:11:05.466785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:05.466749    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a71c5f7-5c2a-4735-848d-9706ff98f961" path="/var/lib/kubelet/pods/2a71c5f7-5c2a-4735-848d-9706ff98f961/volumes"
Nov 02 08:11:14.996116 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:11:14.996071    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:11:17.045681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:17.045645    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"]
Nov 02 08:11:17.058038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:17.058009    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289f0b6d-pwbgs"]
Nov 02 08:11:17.470381 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:17.470316    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32ebe67-ad83-407b-b75b-6fc41e53b807" path="/var/lib/kubelet/pods/f32ebe67-ad83-407b-b75b-6fc41e53b807/volumes"
Nov 02 08:11:31.042308 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:31.042221    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"]
Nov 02 08:11:31.046739 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:31.046712    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e35aec7-bvk7f"]
Nov 02 08:11:31.468942 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:31.468884    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af" path="/var/lib/kubelet/pods/f9edc2e6-2c5d-4cb8-b6d4-6595e7b472af/volumes"
Nov 02 08:11:44.083434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:44.083385    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"]
Nov 02 08:11:44.107483 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:44.107444    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-297e7586-p6qvl"]
Nov 02 08:11:45.146869 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:11:45.146824    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:11:45.469259 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:45.469215    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e88dc09f-e44c-4b86-9f28-3b27722ac761" path="/var/lib/kubelet/pods/e88dc09f-e44c-4b86-9f28-3b27722ac761/volumes"
Nov 02 08:11:57.664459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.664417    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"]
Nov 02 08:11:57.669992 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.669966    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:11:57.688674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.688641    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"]
Nov 02 08:11:57.694639 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.694613    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:11:57.694823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.694648    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:11:57.694823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.694734    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:11:57.696791 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.696744    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:11:57.790390 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.790358    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc53711-tf5rr\" (UID: \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:11:57.790575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.790409    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc53711-tf5rr\" (UID: \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:11:57.790575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.790522    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7f94\" (UniqueName: \"kubernetes.io/projected/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-kube-api-access-f7f94\") pod \"maas-api-revoke-sub-ebc53711-tf5rr\" (UID: \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:11:57.891973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.891934    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc53711-tf5rr\" (UID: \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:11:57.892176 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.892016    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc53711-tf5rr\" (UID: \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:11:57.892176 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.892104    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7f94\" (UniqueName: \"kubernetes.io/projected/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-kube-api-access-f7f94\") pod \"maas-api-revoke-sub-ebc53711-tf5rr\" (UID: \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:11:57.892735 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.892711    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebc53711-tf5rr\" (UID: \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:11:57.894432 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.894405    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebc53711-tf5rr\" (UID: \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:11:57.916737 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.916680    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7f94\" (UniqueName: \"kubernetes.io/projected/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-kube-api-access-f7f94\") pod \"maas-api-revoke-sub-ebc53711-tf5rr\" (UID: \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:11:57.987091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:57.987065    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:11:58.298654 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:11:58.298616    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb5c044_7cf5_45b3_ac2d_28ced2e72686.slice/crio-dad98df493f3c963f40a15e64364d9f764e72c991c9127c1eb3c40a8c7c976e9 WatchSource:0}: Error finding container dad98df493f3c963f40a15e64364d9f764e72c991c9127c1eb3c40a8c7c976e9: Status 404 returned error can't find the container with id dad98df493f3c963f40a15e64364d9f764e72c991c9127c1eb3c40a8c7c976e9
Nov 02 08:11:58.299679 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:58.299655    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"]
Nov 02 08:11:58.570045 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:58.570011    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr" event={"ID":"ddb5c044-7cf5-45b3-ac2d-28ced2e72686","Type":"ContainerStarted","Data":"dad98df493f3c963f40a15e64364d9f764e72c991c9127c1eb3c40a8c7c976e9"}
Nov 02 08:11:59.580486 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:59.580444    4898 generic.go:356] "Generic (PLEG): container finished" podID="ddb5c044-7cf5-45b3-ac2d-28ced2e72686" containerID="9fd4672ca574d50529f3bbc4582f59b196af9f9012ed582a869389308a1364f7" exitCode=0
Nov 02 08:11:59.580984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:11:59.580518    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr" event={"ID":"ddb5c044-7cf5-45b3-ac2d-28ced2e72686","Type":"ContainerDied","Data":"9fd4672ca574d50529f3bbc4582f59b196af9f9012ed582a869389308a1364f7"}
Nov 02 08:12:00.460142 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.460103    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"]
Nov 02 08:12:00.464998 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.464977    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:00.482021 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.481992    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"]
Nov 02 08:12:00.631084 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.631051    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43d3a354-zjj7t\" (UID: \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:00.631475 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.631166    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzh6j\" (UniqueName: \"kubernetes.io/projected/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-kube-api-access-nzh6j\") pod \"maas-api-revoke-sub-43d3a354-zjj7t\" (UID: \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:00.631475 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.631263    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43d3a354-zjj7t\" (UID: \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:00.733519 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.733483    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43d3a354-zjj7t\" (UID: \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:00.733661 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.733595    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzh6j\" (UniqueName: \"kubernetes.io/projected/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-kube-api-access-nzh6j\") pod \"maas-api-revoke-sub-43d3a354-zjj7t\" (UID: \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:00.733748 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.733735    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43d3a354-zjj7t\" (UID: \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:00.734616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.734586    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43d3a354-zjj7t\" (UID: \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:00.735979 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.735954    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43d3a354-zjj7t\" (UID: \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:00.757537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.757514    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzh6j\" (UniqueName: \"kubernetes.io/projected/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-kube-api-access-nzh6j\") pod \"maas-api-revoke-sub-43d3a354-zjj7t\" (UID: \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:00.780219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.780158    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:00.794189 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.794115    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:12:00.942608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.941195    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-kube-api-access-f7f94\" (UniqueName: \"kubernetes.io/projected/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-kube-api-access-f7f94\") pod \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\" (UID: \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\") "
Nov 02 08:12:00.942608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.941263    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-openshift-ca\") pod \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\" (UID: \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\") "
Nov 02 08:12:00.942608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.941314    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-xks-ca\") pod \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\" (UID: \"ddb5c044-7cf5-45b3-ac2d-28ced2e72686\") "
Nov 02 08:12:00.943505 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.943452    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-openshift-ca" pod "ddb5c044-7cf5-45b3-ac2d-28ced2e72686" (UID: "ddb5c044-7cf5-45b3-ac2d-28ced2e72686"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:12:00.955913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.953027    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-kube-api-access-f7f94" pod "ddb5c044-7cf5-45b3-ac2d-28ced2e72686" (UID: "ddb5c044-7cf5-45b3-ac2d-28ced2e72686"). InnerVolumeSpecName "kube-api-access-f7f94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:00.971957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:00.970725    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-xks-ca" pod "ddb5c044-7cf5-45b3-ac2d-28ced2e72686" (UID: "ddb5c044-7cf5-45b3-ac2d-28ced2e72686"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:12:01.044043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:01.043960    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:01.044043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:01.043992    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:01.044043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:01.044002    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7f94\" (UniqueName: \"kubernetes.io/projected/ddb5c044-7cf5-45b3-ac2d-28ced2e72686-kube-api-access-f7f94\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:01.094412 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:12:01.094376    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3b9f44_fd12_4500_84cd_76fbf8b0e20a.slice/crio-e92d0fb5e572a56bb350a91376bb5c5661802ab0c4c92e1026ec1ef747c12ca9 WatchSource:0}: Error finding container e92d0fb5e572a56bb350a91376bb5c5661802ab0c4c92e1026ec1ef747c12ca9: Status 404 returned error can't find the container with id e92d0fb5e572a56bb350a91376bb5c5661802ab0c4c92e1026ec1ef747c12ca9
Nov 02 08:12:01.097574 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:01.097544    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"]
Nov 02 08:12:01.602655 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:01.602626    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"
Nov 02 08:12:01.602818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:01.602648    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr" event={"ID":"ddb5c044-7cf5-45b3-ac2d-28ced2e72686","Type":"ContainerDied","Data":"dad98df493f3c963f40a15e64364d9f764e72c991c9127c1eb3c40a8c7c976e9"}
Nov 02 08:12:01.602818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:01.602709    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dad98df493f3c963f40a15e64364d9f764e72c991c9127c1eb3c40a8c7c976e9"
Nov 02 08:12:01.605515 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:01.605062    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t" event={"ID":"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a","Type":"ContainerStarted","Data":"d65c9ddeaae329195ce23a275b1750e6a1bfbd610285fe1870503d4f24f0339b"}
Nov 02 08:12:01.605515 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:01.605115    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t" event={"ID":"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a","Type":"ContainerStarted","Data":"e92d0fb5e572a56bb350a91376bb5c5661802ab0c4c92e1026ec1ef747c12ca9"}
Nov 02 08:12:01.648221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:01.648163    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t" podStartSLOduration=1.648141632 podStartE2EDuration="1.648141632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:01.645411687 +0000 UTC m=+2872.335738689" watchObservedRunningTime="2026-11-02 08:12:01.648141632 +0000 UTC m=+2872.338468596"
Nov 02 08:12:02.033473 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:02.033383    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"]
Nov 02 08:12:02.038039 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:02.038015    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4f0760-j4flw"]
Nov 02 08:12:02.655283 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:02.655243    4898 generic.go:356] "Generic (PLEG): container finished" podID="1d3b9f44-fd12-4500-84cd-76fbf8b0e20a" containerID="d65c9ddeaae329195ce23a275b1750e6a1bfbd610285fe1870503d4f24f0339b" exitCode=0
Nov 02 08:12:02.655708 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:02.655399    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t" event={"ID":"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a","Type":"ContainerDied","Data":"d65c9ddeaae329195ce23a275b1750e6a1bfbd610285fe1870503d4f24f0339b"}
Nov 02 08:12:03.468112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:03.468075    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d90515-1225-40ec-ae96-4440767ceeb7" path="/var/lib/kubelet/pods/14d90515-1225-40ec-ae96-4440767ceeb7/volumes"
Nov 02 08:12:03.868136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:03.868106    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:03.984216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:03.984133    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-xks-ca\") pod \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\" (UID: \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\") "
Nov 02 08:12:03.984371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:03.984231    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-openshift-ca\") pod \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\" (UID: \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\") "
Nov 02 08:12:03.984491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:03.984463    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-kube-api-access-nzh6j\" (UniqueName: \"kubernetes.io/projected/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-kube-api-access-nzh6j\") pod \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\" (UID: \"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a\") "
Nov 02 08:12:03.984665 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:03.984632    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-openshift-ca" pod "1d3b9f44-fd12-4500-84cd-76fbf8b0e20a" (UID: "1d3b9f44-fd12-4500-84cd-76fbf8b0e20a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:12:03.985230 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:03.985203    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:03.986560 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:03.986533    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-xks-ca" pod "1d3b9f44-fd12-4500-84cd-76fbf8b0e20a" (UID: "1d3b9f44-fd12-4500-84cd-76fbf8b0e20a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:12:03.986646 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:03.986608    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-kube-api-access-nzh6j" pod "1d3b9f44-fd12-4500-84cd-76fbf8b0e20a" (UID: "1d3b9f44-fd12-4500-84cd-76fbf8b0e20a"). InnerVolumeSpecName "kube-api-access-nzh6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:04.086729 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:04.086692    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:04.086729 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:04.086725    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzh6j\" (UniqueName: \"kubernetes.io/projected/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a-kube-api-access-nzh6j\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:04.675996 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:04.675953    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t" event={"ID":"1d3b9f44-fd12-4500-84cd-76fbf8b0e20a","Type":"ContainerDied","Data":"e92d0fb5e572a56bb350a91376bb5c5661802ab0c4c92e1026ec1ef747c12ca9"}
Nov 02 08:12:04.676209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:04.676012    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"
Nov 02 08:12:04.676209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:04.676026    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e92d0fb5e572a56bb350a91376bb5c5661802ab0c4c92e1026ec1ef747c12ca9"
Nov 02 08:12:12.993317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:12.993280    4898 scope.go:122] "RemoveContainer" containerID="fa2e5d4d89a0f80ea62271bbeab0b28b5a1f299c44641068d8ab8a8d190b9e72"
Nov 02 08:12:13.007825 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:13.007794    4898 scope.go:122] "RemoveContainer" containerID="f9911eb9656b24451cdb0967b31c0b817aced8e735c54a3d83e3d0fd2b93986e"
Nov 02 08:12:13.019205 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:13.019180    4898 scope.go:122] "RemoveContainer" containerID="bb3741b5fbaf0d04d1141214c7276f9a5b8ce52e9a63967e099c80523c4b1e55"
Nov 02 08:12:13.031197 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:13.031172    4898 scope.go:122] "RemoveContainer" containerID="1d08934ddfac5213d129c6bf1e229221733bc633328e7edf6ce052c70577fc19"
Nov 02 08:12:14.031724 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.031681    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"]
Nov 02 08:12:14.043675 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.043641    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07691d98-zl9hm"]
Nov 02 08:12:14.687840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.687803    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"]
Nov 02 08:12:14.688613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.688595    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1d3b9f44-fd12-4500-84cd-76fbf8b0e20a" containerName="revoke-keys"
Nov 02 08:12:14.688670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.688615    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3b9f44-fd12-4500-84cd-76fbf8b0e20a" containerName="revoke-keys"
Nov 02 08:12:14.688670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.688638    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ddb5c044-7cf5-45b3-ac2d-28ced2e72686" containerName="revoke-keys"
Nov 02 08:12:14.688670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.688644    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb5c044-7cf5-45b3-ac2d-28ced2e72686" containerName="revoke-keys"
Nov 02 08:12:14.688763 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.688747    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb5c044-7cf5-45b3-ac2d-28ced2e72686" containerName="revoke-keys"
Nov 02 08:12:14.688799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.688765    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3b9f44-fd12-4500-84cd-76fbf8b0e20a" containerName="revoke-keys"
Nov 02 08:12:14.694317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.694296    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:14.720335 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.720303    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"agc-cred-079df9cbf089\""
Nov 02 08:12:14.720780 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.720760    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"openshift-service-ca.crt\""
Nov 02 08:12:14.721156 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.721136    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-71a41c\""
Nov 02 08:12:14.721377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.721363    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"kube-root-ca.crt\""
Nov 02 08:12:14.723385 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.721595    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"payload-processing-external-model-e2e--99e06747-dockercfg-bxgtl\""
Nov 02 08:12:14.733543 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.733512    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"routing-overlay\""
Nov 02 08:12:14.772413 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.768809    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"]
Nov 02 08:12:14.888918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.883696    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:14.888918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.883763    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:14.888918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.883972    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbdpp\" (UniqueName: \"kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-kube-api-access-kbdpp\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:14.888918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.884063    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:15.014266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.998447    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:15.014266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:14.998730    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbdpp\" (UniqueName: \"kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-kube-api-access-kbdpp\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:15.033043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.033013    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:15.036174 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.036123    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:15.037278 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.037246    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:15.038232 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.038206    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:15.050205 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:12:15.050166    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:12:15.053465 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.053397    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:15.057968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.055287    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbdpp\" (UniqueName: \"kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-kube-api-access-kbdpp\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:15.354662 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.354567    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:15.473558 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.473525    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e2015d-3712-43ef-bd55-8f42af1dac59" path="/var/lib/kubelet/pods/84e2015d-3712-43ef-bd55-8f42af1dac59/volumes"
Nov 02 08:12:15.755064 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:12:15.755032    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab522265_0276_413e_99f0_c8266c417bd5.slice/crio-8c4688e68ae5a2bac9cb084c3f87c9f87c2f44e278c86564b951024fa3bebe1d WatchSource:0}: Error finding container 8c4688e68ae5a2bac9cb084c3f87c9f87c2f44e278c86564b951024fa3bebe1d: Status 404 returned error can't find the container with id 8c4688e68ae5a2bac9cb084c3f87c9f87c2f44e278c86564b951024fa3bebe1d
Nov 02 08:12:15.756160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.756130    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"]
Nov 02 08:12:15.761227 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.761199    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc4454848-958cf"]
Nov 02 08:12:15.767707 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.767631    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:15.783118 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.782861    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c68363570711\""
Nov 02 08:12:15.783118 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.782880    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-t5x4n\""
Nov 02 08:12:15.784723 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.784701    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc4454848-958cf"]
Nov 02 08:12:15.788549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.788522    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 08:12:15.788662 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.788637    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 08:12:15.788662 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.788641    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 08:12:15.788780 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.788740    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 08:12:15.854117 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.854089    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-plugins-config-volume\") pod \"payload-processing-external-model-5bc4454848-958cf\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:15.854295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.854277    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcm47\" (UniqueName: \"kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-kube-api-access-lcm47\") pod \"payload-processing-external-model-5bc4454848-958cf\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:15.854374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.854311    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-routing-overlay-volume\") pod \"payload-processing-external-model-5bc4454848-958cf\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:15.854492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.854462    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-provider-credentials\") pod \"payload-processing-external-model-5bc4454848-958cf\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:15.969911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.966056    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcm47\" (UniqueName: \"kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-kube-api-access-lcm47\") pod \"payload-processing-external-model-5bc4454848-958cf\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:15.969911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.966112    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-routing-overlay-volume\") pod \"payload-processing-external-model-5bc4454848-958cf\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:15.969911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.966204    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-provider-credentials\") pod \"payload-processing-external-model-5bc4454848-958cf\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:15.969911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.966393    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-plugins-config-volume\") pod \"payload-processing-external-model-5bc4454848-958cf\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:15.969911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.967457    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-plugins-config-volume\") pod \"payload-processing-external-model-5bc4454848-958cf\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:15.969911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.968420    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-routing-overlay-volume\") pod \"payload-processing-external-model-5bc4454848-958cf\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:15.986628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:15.986595    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-provider-credentials\") pod \"payload-processing-external-model-5bc4454848-958cf\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:16.046649 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:16.046553    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt" event={"ID":"ab522265-0276-413e-99f0-c8266c417bd5","Type":"ContainerStarted","Data":"aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44"}
Nov 02 08:12:16.047272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:16.047240    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt" event={"ID":"ab522265-0276-413e-99f0-c8266c417bd5","Type":"ContainerStarted","Data":"8c4688e68ae5a2bac9cb084c3f87c9f87c2f44e278c86564b951024fa3bebe1d"}
Nov 02 08:12:16.048602 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:16.048578    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:16.071717 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:16.071645    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcm47\" (UniqueName: \"kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-kube-api-access-lcm47\") pod \"payload-processing-external-model-5bc4454848-958cf\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") " pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:16.104532 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:16.104231    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:16.109961 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:16.109874    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt" podStartSLOduration=2.109858227 podStartE2EDuration="2.109858227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:16.105068745 +0000 UTC m=+2886.795395739" watchObservedRunningTime="2026-11-02 08:12:16.109858227 +0000 UTC m=+2886.800185169"
Nov 02 08:12:16.444414 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:12:16.444378    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd650b2f9_4d82_4b05_ad29_b159c45a5c77.slice/crio-21bf3807a442ce2b4242097d0cae5c263b69432b8ebe70361bb471471de7bec7 WatchSource:0}: Error finding container 21bf3807a442ce2b4242097d0cae5c263b69432b8ebe70361bb471471de7bec7: Status 404 returned error can't find the container with id 21bf3807a442ce2b4242097d0cae5c263b69432b8ebe70361bb471471de7bec7
Nov 02 08:12:16.445339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:16.445315    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc4454848-958cf"]
Nov 02 08:12:17.056518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:17.056479    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf" event={"ID":"d650b2f9-4d82-4b05-ad29-b159c45a5c77","Type":"ContainerStarted","Data":"b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810"}
Nov 02 08:12:17.057030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:17.056539    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf" event={"ID":"d650b2f9-4d82-4b05-ad29-b159c45a5c77","Type":"ContainerStarted","Data":"21bf3807a442ce2b4242097d0cae5c263b69432b8ebe70361bb471471de7bec7"}
Nov 02 08:12:17.057030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:17.056574    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:18.816612 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:18.816550    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf" podStartSLOduration=3.816535781 podStartE2EDuration="3.816535781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:17.102361214 +0000 UTC m=+2887.792688179" watchObservedRunningTime="2026-11-02 08:12:18.816535781 +0000 UTC m=+2889.506862764"
Nov 02 08:12:18.818822 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:18.818794    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc4454848-958cf"]
Nov 02 08:12:24.074758 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.074686    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf" podUID="d650b2f9-4d82-4b05-ad29-b159c45a5c77" containerName="payload-processing" containerID="cri-o://b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810" gracePeriod=25
Nov 02 08:12:24.376048 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.376002    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:24.484595 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.484556    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-plugins-config-volume\") pod \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") "
Nov 02 08:12:24.484776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.484732    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-provider-credentials\") pod \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") "
Nov 02 08:12:24.484821 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.484779    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-routing-overlay-volume\") pod \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") "
Nov 02 08:12:24.484953 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.484935    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-kube-api-access-lcm47\" (UniqueName: \"kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-kube-api-access-lcm47\") pod \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\" (UID: \"d650b2f9-4d82-4b05-ad29-b159c45a5c77\") "
Nov 02 08:12:24.485244 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.485210    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-routing-overlay-volume" pod "d650b2f9-4d82-4b05-ad29-b159c45a5c77" (UID: "d650b2f9-4d82-4b05-ad29-b159c45a5c77"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:12:24.485379 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.485251    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-plugins-config-volume" pod "d650b2f9-4d82-4b05-ad29-b159c45a5c77" (UID: "d650b2f9-4d82-4b05-ad29-b159c45a5c77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:12:24.485613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.485590    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:24.485684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.485617    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d650b2f9-4d82-4b05-ad29-b159c45a5c77-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:24.487272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.487237    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-provider-credentials" pod "d650b2f9-4d82-4b05-ad29-b159c45a5c77" (UID: "d650b2f9-4d82-4b05-ad29-b159c45a5c77"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:24.498033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.498009    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-kube-api-access-lcm47" pod "d650b2f9-4d82-4b05-ad29-b159c45a5c77" (UID: "d650b2f9-4d82-4b05-ad29-b159c45a5c77"). InnerVolumeSpecName "kube-api-access-lcm47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:24.586984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.586910    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:24.586984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:24.586944    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcm47\" (UniqueName: \"kubernetes.io/projected/d650b2f9-4d82-4b05-ad29-b159c45a5c77-kube-api-access-lcm47\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:25.069915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.069538    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"]
Nov 02 08:12:25.094908 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.094729    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db8bd722-msg4j"]
Nov 02 08:12:25.170914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.169958    4898 generic.go:356] "Generic (PLEG): container finished" podID="d650b2f9-4d82-4b05-ad29-b159c45a5c77" containerID="b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810" exitCode=0
Nov 02 08:12:25.170914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.170099    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf" event={"ID":"d650b2f9-4d82-4b05-ad29-b159c45a5c77","Type":"ContainerDied","Data":"b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810"}
Nov 02 08:12:25.170914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.170156    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf" event={"ID":"d650b2f9-4d82-4b05-ad29-b159c45a5c77","Type":"ContainerDied","Data":"21bf3807a442ce2b4242097d0cae5c263b69432b8ebe70361bb471471de7bec7"}
Nov 02 08:12:25.170914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.170188    4898 scope.go:122] "RemoveContainer" containerID="b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810"
Nov 02 08:12:25.170914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.170369    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc4454848-958cf"
Nov 02 08:12:25.238602 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.238570    4898 scope.go:122] "RemoveContainer" containerID="b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810"
Nov 02 08:12:25.239200 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:12:25.239156    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810\": container with ID starting with b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810 not found: ID does not exist" containerID="b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810"
Nov 02 08:12:25.239319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.239198    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810"} err="failed to get container status \"b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810\": rpc error: code = NotFound desc = could not find container \"b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810\": container with ID starting with b688921cbb31faf774a9092a87e07220eb4c9994b2bb9ee0db1737cc0f6fb810 not found: ID does not exist"
Nov 02 08:12:25.267926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.266393    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc4454848-958cf"]
Nov 02 08:12:25.277185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.276396    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc4454848-958cf"]
Nov 02 08:12:25.469590 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.469552    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13c1f64-b43a-4473-a587-7fe96fa953f3" path="/var/lib/kubelet/pods/a13c1f64-b43a-4473-a587-7fe96fa953f3/volumes"
Nov 02 08:12:25.470129 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:25.470111    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d650b2f9-4d82-4b05-ad29-b159c45a5c77" path="/var/lib/kubelet/pods/d650b2f9-4d82-4b05-ad29-b159c45a5c77/volumes"
Nov 02 08:12:28.069536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:28.069506    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:12:38.033031 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:38.032986    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"]
Nov 02 08:12:38.046005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:38.045964    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863682c5-7j26k"]
Nov 02 08:12:38.124270 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:38.124214    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:12:38.124607 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:38.124570    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:12:39.469411 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:39.469369    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65bab501-f9b8-4536-9ee4-b6a34a814dc1" path="/var/lib/kubelet/pods/65bab501-f9b8-4536-9ee4-b6a34a814dc1/volumes"
Nov 02 08:12:44.990318 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:12:44.990270    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:12:50.656899 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.656845    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"]
Nov 02 08:12:50.658054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.658033    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d650b2f9-4d82-4b05-ad29-b159c45a5c77" containerName="payload-processing"
Nov 02 08:12:50.658128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.658055    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d650b2f9-4d82-4b05-ad29-b159c45a5c77" containerName="payload-processing"
Nov 02 08:12:50.658234 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.658220    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d650b2f9-4d82-4b05-ad29-b159c45a5c77" containerName="payload-processing"
Nov 02 08:12:50.662502 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.662481    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:50.680206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.680171    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"]
Nov 02 08:12:50.684251 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.684218    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:12:50.684251 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.684230    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:12:50.684466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.684333    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:12:50.694097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.694065    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:12:50.711018 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.710979    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ca4ac6-x9g2g\" (UID: \"e373491d-f04b-4457-9b15-00131cf2b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:50.711223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.711035    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ca4ac6-x9g2g\" (UID: \"e373491d-f04b-4457-9b15-00131cf2b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:50.711223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.711160    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676rh\" (UniqueName: \"kubernetes.io/projected/e373491d-f04b-4457-9b15-00131cf2b58e-kube-api-access-676rh\") pod \"maas-api-revoke-sub-73ca4ac6-x9g2g\" (UID: \"e373491d-f04b-4457-9b15-00131cf2b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:50.812416 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.812374    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676rh\" (UniqueName: \"kubernetes.io/projected/e373491d-f04b-4457-9b15-00131cf2b58e-kube-api-access-676rh\") pod \"maas-api-revoke-sub-73ca4ac6-x9g2g\" (UID: \"e373491d-f04b-4457-9b15-00131cf2b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:50.812634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.812549    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ca4ac6-x9g2g\" (UID: \"e373491d-f04b-4457-9b15-00131cf2b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:50.812634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.812573    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ca4ac6-x9g2g\" (UID: \"e373491d-f04b-4457-9b15-00131cf2b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:50.813267 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.813244    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ca4ac6-x9g2g\" (UID: \"e373491d-f04b-4457-9b15-00131cf2b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:50.815570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.815543    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ca4ac6-x9g2g\" (UID: \"e373491d-f04b-4457-9b15-00131cf2b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:50.835615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.835583    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-676rh\" (UniqueName: \"kubernetes.io/projected/e373491d-f04b-4457-9b15-00131cf2b58e-kube-api-access-676rh\") pod \"maas-api-revoke-sub-73ca4ac6-x9g2g\" (UID: \"e373491d-f04b-4457-9b15-00131cf2b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:50.986469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:50.986377    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:51.308358 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:12:51.308327    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode373491d_f04b_4457_9b15_00131cf2b58e.slice/crio-b2aad70a12e3780ff55c7d338315c4137c83974bb5a0e36ee86a6f3ada9dfedf WatchSource:0}: Error finding container b2aad70a12e3780ff55c7d338315c4137c83974bb5a0e36ee86a6f3ada9dfedf: Status 404 returned error can't find the container with id b2aad70a12e3780ff55c7d338315c4137c83974bb5a0e36ee86a6f3ada9dfedf
Nov 02 08:12:51.311229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:51.311195    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"]
Nov 02 08:12:51.725266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:51.725178    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g" event={"ID":"e373491d-f04b-4457-9b15-00131cf2b58e","Type":"ContainerStarted","Data":"21658e22d41edc0acc6ae1f1de8f1437fa20ff4dab728a2c744b47437a47e649"}
Nov 02 08:12:51.725266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:51.725246    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g" event={"ID":"e373491d-f04b-4457-9b15-00131cf2b58e","Type":"ContainerStarted","Data":"b2aad70a12e3780ff55c7d338315c4137c83974bb5a0e36ee86a6f3ada9dfedf"}
Nov 02 08:12:51.765660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:51.765607    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g" podStartSLOduration=1.765594812 podStartE2EDuration="1.765594812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:12:51.764791883 +0000 UTC m=+2922.455118847" watchObservedRunningTime="2026-11-02 08:12:51.765594812 +0000 UTC m=+2922.455921776"
Nov 02 08:12:52.760533 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:52.760498    4898 generic.go:356] "Generic (PLEG): container finished" podID="e373491d-f04b-4457-9b15-00131cf2b58e" containerID="21658e22d41edc0acc6ae1f1de8f1437fa20ff4dab728a2c744b47437a47e649" exitCode=0
Nov 02 08:12:52.760993 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:52.760579    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g" event={"ID":"e373491d-f04b-4457-9b15-00131cf2b58e","Type":"ContainerDied","Data":"21658e22d41edc0acc6ae1f1de8f1437fa20ff4dab728a2c744b47437a47e649"}
Nov 02 08:12:53.956233 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:53.956197    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:53.998599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:53.998563    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-openshift-ca\") pod \"e373491d-f04b-4457-9b15-00131cf2b58e\" (UID: \"e373491d-f04b-4457-9b15-00131cf2b58e\") "
Nov 02 08:12:53.998777 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:53.998722    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e373491d-f04b-4457-9b15-00131cf2b58e-kube-api-access-676rh\" (UniqueName: \"kubernetes.io/projected/e373491d-f04b-4457-9b15-00131cf2b58e-kube-api-access-676rh\") pod \"e373491d-f04b-4457-9b15-00131cf2b58e\" (UID: \"e373491d-f04b-4457-9b15-00131cf2b58e\") "
Nov 02 08:12:53.998851 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:53.998781    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-xks-ca\") pod \"e373491d-f04b-4457-9b15-00131cf2b58e\" (UID: \"e373491d-f04b-4457-9b15-00131cf2b58e\") "
Nov 02 08:12:53.998991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:53.998962    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-openshift-ca" pod "e373491d-f04b-4457-9b15-00131cf2b58e" (UID: "e373491d-f04b-4457-9b15-00131cf2b58e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:12:53.999577 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:53.999550    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:54.001084 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:54.001055    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e373491d-f04b-4457-9b15-00131cf2b58e-kube-api-access-676rh" pod "e373491d-f04b-4457-9b15-00131cf2b58e" (UID: "e373491d-f04b-4457-9b15-00131cf2b58e"). InnerVolumeSpecName "kube-api-access-676rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:12:54.001084 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:54.001073    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-xks-ca" pod "e373491d-f04b-4457-9b15-00131cf2b58e" (UID: "e373491d-f04b-4457-9b15-00131cf2b58e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:12:54.101195 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:54.101095    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-676rh\" (UniqueName: \"kubernetes.io/projected/e373491d-f04b-4457-9b15-00131cf2b58e-kube-api-access-676rh\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:54.101195 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:54.101140    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e373491d-f04b-4457-9b15-00131cf2b58e-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:12:54.779516 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:54.779485    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"
Nov 02 08:12:54.779516 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:54.779505    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g" event={"ID":"e373491d-f04b-4457-9b15-00131cf2b58e","Type":"ContainerDied","Data":"b2aad70a12e3780ff55c7d338315c4137c83974bb5a0e36ee86a6f3ada9dfedf"}
Nov 02 08:12:54.779756 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:12:54.779554    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2aad70a12e3780ff55c7d338315c4137c83974bb5a0e36ee86a6f3ada9dfedf"
Nov 02 08:13:01.437570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.437535    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"]
Nov 02 08:13:01.438535 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.438508    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e373491d-f04b-4457-9b15-00131cf2b58e" containerName="revoke-keys"
Nov 02 08:13:01.438535 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.438535    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e373491d-f04b-4457-9b15-00131cf2b58e" containerName="revoke-keys"
Nov 02 08:13:01.438771 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.438756    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e373491d-f04b-4457-9b15-00131cf2b58e" containerName="revoke-keys"
Nov 02 08:13:01.442921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.442899    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:01.468148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.468004    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:13:01.468148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.468059    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:13:01.468148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.468105    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:13:01.468148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.468016    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:13:01.482569 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.482536    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"]
Nov 02 08:13:01.525615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.525578    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6468ca-fckkn\" (UID: \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:01.525804 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.525786    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6468ca-fckkn\" (UID: \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:01.525846 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.525823    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6f5g\" (UniqueName: \"kubernetes.io/projected/97c42383-c2e4-4ddc-89dc-c54ca28b5692-kube-api-access-g6f5g\") pod \"maas-api-revoke-sub-1b6468ca-fckkn\" (UID: \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:01.627441 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.627408    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6f5g\" (UniqueName: \"kubernetes.io/projected/97c42383-c2e4-4ddc-89dc-c54ca28b5692-kube-api-access-g6f5g\") pod \"maas-api-revoke-sub-1b6468ca-fckkn\" (UID: \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:01.627658 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.627547    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6468ca-fckkn\" (UID: \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:01.627844 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.627819    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6468ca-fckkn\" (UID: \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:01.628539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.628513    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6468ca-fckkn\" (UID: \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:01.630097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.630076    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6468ca-fckkn\" (UID: \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:01.651305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.651270    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6f5g\" (UniqueName: \"kubernetes.io/projected/97c42383-c2e4-4ddc-89dc-c54ca28b5692-kube-api-access-g6f5g\") pod \"maas-api-revoke-sub-1b6468ca-fckkn\" (UID: \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:01.760094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:01.759981    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:02.073996 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:13:02.073962    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c42383_c2e4_4ddc_89dc_c54ca28b5692.slice/crio-d428207ad4aa7d01d6b104cc36ee590644b80bd7cac84f1be8faf2a2eb388e71 WatchSource:0}: Error finding container d428207ad4aa7d01d6b104cc36ee590644b80bd7cac84f1be8faf2a2eb388e71: Status 404 returned error can't find the container with id d428207ad4aa7d01d6b104cc36ee590644b80bd7cac84f1be8faf2a2eb388e71
Nov 02 08:13:02.074815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:02.074787    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"]
Nov 02 08:13:02.873756 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:02.873714    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn" event={"ID":"97c42383-c2e4-4ddc-89dc-c54ca28b5692","Type":"ContainerStarted","Data":"269af8961e02fced5a07a31439b82c2da88fc8383ed90ee4d0e6ab56e96cbfbd"}
Nov 02 08:13:02.874315 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:02.873778    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn" event={"ID":"97c42383-c2e4-4ddc-89dc-c54ca28b5692","Type":"ContainerStarted","Data":"d428207ad4aa7d01d6b104cc36ee590644b80bd7cac84f1be8faf2a2eb388e71"}
Nov 02 08:13:02.918245 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:02.918188    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn" podStartSLOduration=1.918172508 podStartE2EDuration="1.918172508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:02.913350161 +0000 UTC m=+2933.603677149" watchObservedRunningTime="2026-11-02 08:13:02.918172508 +0000 UTC m=+2933.608499471"
Nov 02 08:13:03.884301 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:03.884261    4898 generic.go:356] "Generic (PLEG): container finished" podID="97c42383-c2e4-4ddc-89dc-c54ca28b5692" containerID="269af8961e02fced5a07a31439b82c2da88fc8383ed90ee4d0e6ab56e96cbfbd" exitCode=0
Nov 02 08:13:03.884763 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:03.884342    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn" event={"ID":"97c42383-c2e4-4ddc-89dc-c54ca28b5692","Type":"ContainerDied","Data":"269af8961e02fced5a07a31439b82c2da88fc8383ed90ee4d0e6ab56e96cbfbd"}
Nov 02 08:13:05.079559 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.079532    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:05.093829 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.093757    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-openshift-ca\") pod \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\" (UID: \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\") "
Nov 02 08:13:05.093829 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.093814    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-xks-ca\") pod \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\" (UID: \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\") "
Nov 02 08:13:05.094235 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.094206    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-openshift-ca" pod "97c42383-c2e4-4ddc-89dc-c54ca28b5692" (UID: "97c42383-c2e4-4ddc-89dc-c54ca28b5692"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:05.094497 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.094477    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:05.096547 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.096520    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-xks-ca" pod "97c42383-c2e4-4ddc-89dc-c54ca28b5692" (UID: "97c42383-c2e4-4ddc-89dc-c54ca28b5692"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:13:05.195580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.195537    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c42383-c2e4-4ddc-89dc-c54ca28b5692-kube-api-access-g6f5g\" (UniqueName: \"kubernetes.io/projected/97c42383-c2e4-4ddc-89dc-c54ca28b5692-kube-api-access-g6f5g\") pod \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\" (UID: \"97c42383-c2e4-4ddc-89dc-c54ca28b5692\") "
Nov 02 08:13:05.196332 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.196305    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c42383-c2e4-4ddc-89dc-c54ca28b5692-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:05.197870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.197838    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c42383-c2e4-4ddc-89dc-c54ca28b5692-kube-api-access-g6f5g" pod "97c42383-c2e4-4ddc-89dc-c54ca28b5692" (UID: "97c42383-c2e4-4ddc-89dc-c54ca28b5692"). InnerVolumeSpecName "kube-api-access-g6f5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:05.297685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.297646    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6f5g\" (UniqueName: \"kubernetes.io/projected/97c42383-c2e4-4ddc-89dc-c54ca28b5692-kube-api-access-g6f5g\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:05.906225 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.906190    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"
Nov 02 08:13:05.906397 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.906220    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn" event={"ID":"97c42383-c2e4-4ddc-89dc-c54ca28b5692","Type":"ContainerDied","Data":"d428207ad4aa7d01d6b104cc36ee590644b80bd7cac84f1be8faf2a2eb388e71"}
Nov 02 08:13:05.906397 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:05.906275    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d428207ad4aa7d01d6b104cc36ee590644b80bd7cac84f1be8faf2a2eb388e71"
Nov 02 08:13:10.033884 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:10.033843    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"]
Nov 02 08:13:10.038437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:10.038410    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-580227db-g2krw"]
Nov 02 08:13:11.467842 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:11.467806    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69769763-f064-474f-9ddf-300b7f499a53" path="/var/lib/kubelet/pods/69769763-f064-474f-9ddf-300b7f499a53/volumes"
Nov 02 08:13:13.108917 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:13.101304    4898 scope.go:122] "RemoveContainer" containerID="39a65db83c9c721554da47c20dd290b51d55db4e3f58fa74429464e3146dfb21"
Nov 02 08:13:13.141509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:13.141479    4898 scope.go:122] "RemoveContainer" containerID="3be2d174a658da07e6eaba3d7c1e5d849e3d92deaf31e5d7018d9f5008c86c51"
Nov 02 08:13:13.152238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:13.152214    4898 scope.go:122] "RemoveContainer" containerID="d05a22630e09fa9d44ff1d14e04699c5dc12ff354bac0061706691701beb9800"
Nov 02 08:13:13.162531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:13.162509    4898 scope.go:122] "RemoveContainer" containerID="ad37446093ab5fc3560e7b22ab66675d1837f8cc6234c3b67442a74a43032ef4"
Nov 02 08:13:13.177779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:13.177749    4898 scope.go:122] "RemoveContainer" containerID="2179f10f52358457e5e1de3537895af4498b948f6423caaac5a6e696c24aa99c"
Nov 02 08:13:13.780856 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:13.780820    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"]
Nov 02 08:13:14.998617 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:13:14.998574    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:13:18.781182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:18.781140    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p" podUID="cd38daae-cf3a-47cd-9055-f60195e3fa92" containerName="payload-processing" containerID="cri-o://02f29ed590760cefbd91568c91e71c0cd29371b3663d6011210089d730f1d928" gracePeriod=25
Nov 02 08:13:19.041040 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.040768    4898 generic.go:356] "Generic (PLEG): container finished" podID="cd38daae-cf3a-47cd-9055-f60195e3fa92" containerID="02f29ed590760cefbd91568c91e71c0cd29371b3663d6011210089d730f1d928" exitCode=0
Nov 02 08:13:19.041040 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.040942    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p" event={"ID":"cd38daae-cf3a-47cd-9055-f60195e3fa92","Type":"ContainerDied","Data":"02f29ed590760cefbd91568c91e71c0cd29371b3663d6011210089d730f1d928"}
Nov 02 08:13:19.102412 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.102381    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:13:19.255805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.255767    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-routing-overlay-volume\") pod \"cd38daae-cf3a-47cd-9055-f60195e3fa92\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") "
Nov 02 08:13:19.256054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.256036    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-provider-credentials\") pod \"cd38daae-cf3a-47cd-9055-f60195e3fa92\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") "
Nov 02 08:13:19.256163 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.256099    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-plugins-config-volume\") pod \"cd38daae-cf3a-47cd-9055-f60195e3fa92\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") "
Nov 02 08:13:19.256163 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.256138    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-kube-api-access-w54vh\" (UniqueName: \"kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-kube-api-access-w54vh\") pod \"cd38daae-cf3a-47cd-9055-f60195e3fa92\" (UID: \"cd38daae-cf3a-47cd-9055-f60195e3fa92\") "
Nov 02 08:13:19.256292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.256231    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-routing-overlay-volume" pod "cd38daae-cf3a-47cd-9055-f60195e3fa92" (UID: "cd38daae-cf3a-47cd-9055-f60195e3fa92"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:19.256694 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.256668    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-plugins-config-volume" pod "cd38daae-cf3a-47cd-9055-f60195e3fa92" (UID: "cd38daae-cf3a-47cd-9055-f60195e3fa92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:19.256932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.256914    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:19.256987 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.256938    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd38daae-cf3a-47cd-9055-f60195e3fa92-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:19.260095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.260063    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-provider-credentials" pod "cd38daae-cf3a-47cd-9055-f60195e3fa92" (UID: "cd38daae-cf3a-47cd-9055-f60195e3fa92"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:19.260219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.260134    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-kube-api-access-w54vh" pod "cd38daae-cf3a-47cd-9055-f60195e3fa92" (UID: "cd38daae-cf3a-47cd-9055-f60195e3fa92"). InnerVolumeSpecName "kube-api-access-w54vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:19.326211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.326098    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"]
Nov 02 08:13:19.327009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.326984    4898 cpu_manager.go:414] "containerMap: removing container" podUID="cd38daae-cf3a-47cd-9055-f60195e3fa92" containerName="payload-processing"
Nov 02 08:13:19.327143 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.327010    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd38daae-cf3a-47cd-9055-f60195e3fa92" containerName="payload-processing"
Nov 02 08:13:19.327143 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.327062    4898 cpu_manager.go:414] "containerMap: removing container" podUID="97c42383-c2e4-4ddc-89dc-c54ca28b5692" containerName="revoke-keys"
Nov 02 08:13:19.327143 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.327070    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c42383-c2e4-4ddc-89dc-c54ca28b5692" containerName="revoke-keys"
Nov 02 08:13:19.327301 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.327291    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd38daae-cf3a-47cd-9055-f60195e3fa92" containerName="payload-processing"
Nov 02 08:13:19.327352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.327314    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c42383-c2e4-4ddc-89dc-c54ca28b5692" containerName="revoke-keys"
Nov 02 08:13:19.331370 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.331344    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:19.346460 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.346418    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:13:19.346460 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.346418    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:13:19.346713 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.346435    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:13:19.346713 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.346492    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:13:19.348681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.348651    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"]
Nov 02 08:13:19.359881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.359830    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:19.359881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.359873    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w54vh\" (UniqueName: \"kubernetes.io/projected/cd38daae-cf3a-47cd-9055-f60195e3fa92-kube-api-access-w54vh\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:19.461401 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.461362    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ee127b-7g8r8\" (UID: \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:19.461624 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.461419    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ee127b-7g8r8\" (UID: \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:19.461713 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.461684    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcv5k\" (UniqueName: \"kubernetes.io/projected/033f9df6-5ac0-4d0d-94ed-cc914d7392df-kube-api-access-tcv5k\") pod \"maas-api-revoke-sub-d4ee127b-7g8r8\" (UID: \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:19.563103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.563052    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ee127b-7g8r8\" (UID: \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:19.563332 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.563111    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ee127b-7g8r8\" (UID: \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:19.563408 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.563352    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcv5k\" (UniqueName: \"kubernetes.io/projected/033f9df6-5ac0-4d0d-94ed-cc914d7392df-kube-api-access-tcv5k\") pod \"maas-api-revoke-sub-d4ee127b-7g8r8\" (UID: \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:19.563968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.563934    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ee127b-7g8r8\" (UID: \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:19.566568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.566539    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ee127b-7g8r8\" (UID: \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:19.586797 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.586717    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcv5k\" (UniqueName: \"kubernetes.io/projected/033f9df6-5ac0-4d0d-94ed-cc914d7392df-kube-api-access-tcv5k\") pod \"maas-api-revoke-sub-d4ee127b-7g8r8\" (UID: \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:19.652293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.652225    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:19.961922 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:13:19.961848    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033f9df6_5ac0_4d0d_94ed_cc914d7392df.slice/crio-c219b13e64f9ab26f53d64dd690a0b7a986e847c84bb310dbafc9a801162b6bb WatchSource:0}: Error finding container c219b13e64f9ab26f53d64dd690a0b7a986e847c84bb310dbafc9a801162b6bb: Status 404 returned error can't find the container with id c219b13e64f9ab26f53d64dd690a0b7a986e847c84bb310dbafc9a801162b6bb
Nov 02 08:13:19.965156 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:19.965128    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"]
Nov 02 08:13:20.032843 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:20.032804    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"]
Nov 02 08:13:20.037598 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:20.037564    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fa503cd-zxhwx"]
Nov 02 08:13:20.051202 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:20.051173    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"
Nov 02 08:13:20.051362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:20.051196    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p" event={"ID":"cd38daae-cf3a-47cd-9055-f60195e3fa92","Type":"ContainerDied","Data":"dada9bfe3ba1e5db78321714a6240a52c96c412495516c1fd3737ea3f63ae7bf"}
Nov 02 08:13:20.051362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:20.051263    4898 scope.go:122] "RemoveContainer" containerID="02f29ed590760cefbd91568c91e71c0cd29371b3663d6011210089d730f1d928"
Nov 02 08:13:20.052773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:20.052747    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8" event={"ID":"033f9df6-5ac0-4d0d-94ed-cc914d7392df","Type":"ContainerStarted","Data":"c219b13e64f9ab26f53d64dd690a0b7a986e847c84bb310dbafc9a801162b6bb"}
Nov 02 08:13:20.108539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:20.108503    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"]
Nov 02 08:13:20.113031 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:20.113008    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c712b1/payload-processing-external-model-e2e-extmod-a-c712b1-5bd5xmg5p"]
Nov 02 08:13:21.072166 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:21.072121    4898 generic.go:356] "Generic (PLEG): container finished" podID="033f9df6-5ac0-4d0d-94ed-cc914d7392df" containerID="b951da923ae84088d01b9eff0f351ed1a1bb7e7b1066ac9f34dd51ac39c0e7ab" exitCode=0
Nov 02 08:13:21.072650 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:21.072201    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8" event={"ID":"033f9df6-5ac0-4d0d-94ed-cc914d7392df","Type":"ContainerDied","Data":"b951da923ae84088d01b9eff0f351ed1a1bb7e7b1066ac9f34dd51ac39c0e7ab"}
Nov 02 08:13:21.469039 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:21.469002    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984f0c6e-8b5d-4599-bc5a-8414b26d21c1" path="/var/lib/kubelet/pods/984f0c6e-8b5d-4599-bc5a-8414b26d21c1/volumes"
Nov 02 08:13:21.469574 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:21.469556    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd38daae-cf3a-47cd-9055-f60195e3fa92" path="/var/lib/kubelet/pods/cd38daae-cf3a-47cd-9055-f60195e3fa92/volumes"
Nov 02 08:13:22.347209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:22.347137    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:22.524304 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:22.524270    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-xks-ca\") pod \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\" (UID: \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\") "
Nov 02 08:13:22.524491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:22.524410    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033f9df6-5ac0-4d0d-94ed-cc914d7392df-kube-api-access-tcv5k\" (UniqueName: \"kubernetes.io/projected/033f9df6-5ac0-4d0d-94ed-cc914d7392df-kube-api-access-tcv5k\") pod \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\" (UID: \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\") "
Nov 02 08:13:22.524491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:22.524458    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-openshift-ca\") pod \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\" (UID: \"033f9df6-5ac0-4d0d-94ed-cc914d7392df\") "
Nov 02 08:13:22.524841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:22.524814    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-openshift-ca" pod "033f9df6-5ac0-4d0d-94ed-cc914d7392df" (UID: "033f9df6-5ac0-4d0d-94ed-cc914d7392df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:22.526386 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:22.526362    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-xks-ca" pod "033f9df6-5ac0-4d0d-94ed-cc914d7392df" (UID: "033f9df6-5ac0-4d0d-94ed-cc914d7392df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:13:22.526533 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:22.526511    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033f9df6-5ac0-4d0d-94ed-cc914d7392df-kube-api-access-tcv5k" pod "033f9df6-5ac0-4d0d-94ed-cc914d7392df" (UID: "033f9df6-5ac0-4d0d-94ed-cc914d7392df"). InnerVolumeSpecName "kube-api-access-tcv5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:22.625837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:22.625750    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcv5k\" (UniqueName: \"kubernetes.io/projected/033f9df6-5ac0-4d0d-94ed-cc914d7392df-kube-api-access-tcv5k\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:22.625837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:22.625786    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:22.625837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:22.625798    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033f9df6-5ac0-4d0d-94ed-cc914d7392df-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:23.095183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:23.095152    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"
Nov 02 08:13:23.095183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:23.095170    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8" event={"ID":"033f9df6-5ac0-4d0d-94ed-cc914d7392df","Type":"ContainerDied","Data":"c219b13e64f9ab26f53d64dd690a0b7a986e847c84bb310dbafc9a801162b6bb"}
Nov 02 08:13:23.095410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:23.095221    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c219b13e64f9ab26f53d64dd690a0b7a986e847c84bb310dbafc9a801162b6bb"
Nov 02 08:13:31.530946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.527244    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"]
Nov 02 08:13:31.532660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.532627    4898 cpu_manager.go:414] "containerMap: removing container" podUID="033f9df6-5ac0-4d0d-94ed-cc914d7392df" containerName="revoke-keys"
Nov 02 08:13:31.532660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.532658    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033f9df6-5ac0-4d0d-94ed-cc914d7392df" containerName="revoke-keys"
Nov 02 08:13:31.532962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.532883    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="033f9df6-5ac0-4d0d-94ed-cc914d7392df" containerName="revoke-keys"
Nov 02 08:13:31.540735 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.537187    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"]
Nov 02 08:13:31.540735 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.537304    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:31.554460 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.554427    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:13:31.554605 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.554438    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:13:31.554677 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.554644    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:13:31.572921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.572675    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:13:31.658924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.648204    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbppq\" (UniqueName: \"kubernetes.io/projected/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-kube-api-access-fbppq\") pod \"maas-api-revoke-sub-a970f776-bxtjw\" (UID: \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:31.658924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.648323    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a970f776-bxtjw\" (UID: \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:31.658924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.648647    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a970f776-bxtjw\" (UID: \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:31.750580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.750539    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbppq\" (UniqueName: \"kubernetes.io/projected/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-kube-api-access-fbppq\") pod \"maas-api-revoke-sub-a970f776-bxtjw\" (UID: \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:31.750813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.750612    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a970f776-bxtjw\" (UID: \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:31.750813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.750788    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a970f776-bxtjw\" (UID: \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:31.751452 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.751427    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a970f776-bxtjw\" (UID: \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:31.753209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.753187    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a970f776-bxtjw\" (UID: \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:31.775719 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.775686    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbppq\" (UniqueName: \"kubernetes.io/projected/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-kube-api-access-fbppq\") pod \"maas-api-revoke-sub-a970f776-bxtjw\" (UID: \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:31.891000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:31.890910    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:32.209802 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:13:32.209763    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b408bd6_3dbf_4045_8ce5_b0e1e9abd268.slice/crio-2480508f44020799db5bd7c5820e3d24b111371709c989c00f2708a698f9ca1f WatchSource:0}: Error finding container 2480508f44020799db5bd7c5820e3d24b111371709c989c00f2708a698f9ca1f: Status 404 returned error can't find the container with id 2480508f44020799db5bd7c5820e3d24b111371709c989c00f2708a698f9ca1f
Nov 02 08:13:32.213132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:32.213039    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"]
Nov 02 08:13:32.251744 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:32.251715    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw" event={"ID":"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268","Type":"ContainerStarted","Data":"2480508f44020799db5bd7c5820e3d24b111371709c989c00f2708a698f9ca1f"}
Nov 02 08:13:33.264715 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:33.264671    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw" event={"ID":"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268","Type":"ContainerStarted","Data":"0ef87bfc62b0d4b09398537fa0717c077a792d9484892ebcd04814932107223d"}
Nov 02 08:13:33.309375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:33.309313    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw" podStartSLOduration=2.30929587 podStartE2EDuration="2.30929587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:33.309098841 +0000 UTC m=+2963.999425824" watchObservedRunningTime="2026-11-02 08:13:33.30929587 +0000 UTC m=+2963.999622834"
Nov 02 08:13:33.867119 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:33.867082    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"]
Nov 02 08:13:33.872087 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:33.872058    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"
Nov 02 08:13:33.887732 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:33.887703    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e271b1-dockercfg-wn827\""
Nov 02 08:13:33.893072 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:33.893039    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"]
Nov 02 08:13:34.097872 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.097594    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e271b1\""
Nov 02 08:13:34.171945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.168793    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"]
Nov 02 08:13:34.175010 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.174518    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"]
Nov 02 08:13:34.175010 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.174604    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"
Nov 02 08:13:34.250409 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.250345    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfht7\" (UniqueName: \"kubernetes.io/projected/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-kube-api-access-cfht7\") pod \"payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq\" (UID: \"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"
Nov 02 08:13:34.250603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.250466    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq\" (UID: \"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"
Nov 02 08:13:34.250603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.250556    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx4mk\" (UniqueName: \"kubernetes.io/projected/77a4db1f-32da-42da-ab59-c959e7812faf-kube-api-access-lx4mk\") pod \"payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp\" (UID: \"77a4db1f-32da-42da-ab59-c959e7812faf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"
Nov 02 08:13:34.250734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.250720    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a4db1f-32da-42da-ab59-c959e7812faf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp\" (UID: \"77a4db1f-32da-42da-ab59-c959e7812faf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"
Nov 02 08:13:34.275906 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.275864    4898 generic.go:356] "Generic (PLEG): container finished" podID="1b408bd6-3dbf-4045-8ce5-b0e1e9abd268" containerID="0ef87bfc62b0d4b09398537fa0717c077a792d9484892ebcd04814932107223d" exitCode=0
Nov 02 08:13:34.276434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.275963    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw" event={"ID":"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268","Type":"ContainerDied","Data":"0ef87bfc62b0d4b09398537fa0717c077a792d9484892ebcd04814932107223d"}
Nov 02 08:13:34.351826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.351787    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a4db1f-32da-42da-ab59-c959e7812faf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp\" (UID: \"77a4db1f-32da-42da-ab59-c959e7812faf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"
Nov 02 08:13:34.352056 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.352023    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfht7\" (UniqueName: \"kubernetes.io/projected/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-kube-api-access-cfht7\") pod \"payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq\" (UID: \"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"
Nov 02 08:13:34.352128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.352064    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq\" (UID: \"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"
Nov 02 08:13:34.352201 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.352137    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4mk\" (UniqueName: \"kubernetes.io/projected/77a4db1f-32da-42da-ab59-c959e7812faf-kube-api-access-lx4mk\") pod \"payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp\" (UID: \"77a4db1f-32da-42da-ab59-c959e7812faf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"
Nov 02 08:13:34.352740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.352715    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a4db1f-32da-42da-ab59-c959e7812faf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp\" (UID: \"77a4db1f-32da-42da-ab59-c959e7812faf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"
Nov 02 08:13:34.352957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.352928    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq\" (UID: \"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"
Nov 02 08:13:34.376835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.376762    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfht7\" (UniqueName: \"kubernetes.io/projected/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-kube-api-access-cfht7\") pod \"payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq\" (UID: \"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"
Nov 02 08:13:34.382431 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.382403    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4mk\" (UniqueName: \"kubernetes.io/projected/77a4db1f-32da-42da-ab59-c959e7812faf-kube-api-access-lx4mk\") pod \"payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp\" (UID: \"77a4db1f-32da-42da-ab59-c959e7812faf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"
Nov 02 08:13:34.487094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.487003    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"
Nov 02 08:13:34.494539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.494511    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"
Nov 02 08:13:34.815336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.815295    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"]
Nov 02 08:13:34.816676 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:13:34.816633    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffef3aaf_476b_4037_a4ac_cfc3c89bfb40.slice/crio-a2db688565355e8e9f338e1c6616506b67dae4e3d59cd7c89f2602324361c495 WatchSource:0}: Error finding container a2db688565355e8e9f338e1c6616506b67dae4e3d59cd7c89f2602324361c495: Status 404 returned error can't find the container with id a2db688565355e8e9f338e1c6616506b67dae4e3d59cd7c89f2602324361c495
Nov 02 08:13:34.836638 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:13:34.836611    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a4db1f_32da_42da_ab59_c959e7812faf.slice/crio-e0d4b39217475833331898831de917a489309d2fd2412a96aaa8a9ac95ab348e WatchSource:0}: Error finding container e0d4b39217475833331898831de917a489309d2fd2412a96aaa8a9ac95ab348e: Status 404 returned error can't find the container with id e0d4b39217475833331898831de917a489309d2fd2412a96aaa8a9ac95ab348e
Nov 02 08:13:34.851849 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:34.851806    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"]
Nov 02 08:13:35.301939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.301683    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp" event={"ID":"77a4db1f-32da-42da-ab59-c959e7812faf","Type":"ContainerStarted","Data":"598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434"}
Nov 02 08:13:35.301939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.301749    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp" event={"ID":"77a4db1f-32da-42da-ab59-c959e7812faf","Type":"ContainerStarted","Data":"e0d4b39217475833331898831de917a489309d2fd2412a96aaa8a9ac95ab348e"}
Nov 02 08:13:35.303578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.302815    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"
Nov 02 08:13:35.305465 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.305421    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq" event={"ID":"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40","Type":"ContainerStarted","Data":"17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60"}
Nov 02 08:13:35.305602 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.305498    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq" event={"ID":"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40","Type":"ContainerStarted","Data":"a2db688565355e8e9f338e1c6616506b67dae4e3d59cd7c89f2602324361c495"}
Nov 02 08:13:35.305602 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.305597    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"
Nov 02 08:13:35.346978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.344797    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp" podStartSLOduration=2.344776889 podStartE2EDuration="2.344776889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:35.343140826 +0000 UTC m=+2966.033467815" watchObservedRunningTime="2026-11-02 08:13:35.344776889 +0000 UTC m=+2966.035103854"
Nov 02 08:13:35.416603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.407816    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq" podStartSLOduration=2.40779547 podStartE2EDuration="2.40779547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:35.401766286 +0000 UTC m=+2966.092093276" watchObservedRunningTime="2026-11-02 08:13:35.40779547 +0000 UTC m=+2966.098122436"
Nov 02 08:13:35.580132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.580051    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:35.679428 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.679385    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-openshift-ca\") pod \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\" (UID: \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\") "
Nov 02 08:13:35.679621 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.679608    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-xks-ca\") pod \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\" (UID: \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\") "
Nov 02 08:13:35.679841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.679811    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-openshift-ca" pod "1b408bd6-3dbf-4045-8ce5-b0e1e9abd268" (UID: "1b408bd6-3dbf-4045-8ce5-b0e1e9abd268"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:35.679841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.679838    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-kube-api-access-fbppq\" (UniqueName: \"kubernetes.io/projected/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-kube-api-access-fbppq\") pod \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\" (UID: \"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268\") "
Nov 02 08:13:35.680485 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.680457    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:35.682508 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.682469    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-xks-ca" pod "1b408bd6-3dbf-4045-8ce5-b0e1e9abd268" (UID: "1b408bd6-3dbf-4045-8ce5-b0e1e9abd268"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:13:35.682591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.682553    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-kube-api-access-fbppq" pod "1b408bd6-3dbf-4045-8ce5-b0e1e9abd268" (UID: "1b408bd6-3dbf-4045-8ce5-b0e1e9abd268"). InnerVolumeSpecName "kube-api-access-fbppq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:35.782774 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.782711    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbppq\" (UniqueName: \"kubernetes.io/projected/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-kube-api-access-fbppq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:35.782774 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:35.782774    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:36.315993 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:36.315962    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"
Nov 02 08:13:36.315993 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:36.315981    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw" event={"ID":"1b408bd6-3dbf-4045-8ce5-b0e1e9abd268","Type":"ContainerDied","Data":"2480508f44020799db5bd7c5820e3d24b111371709c989c00f2708a698f9ca1f"}
Nov 02 08:13:36.316591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:36.316031    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2480508f44020799db5bd7c5820e3d24b111371709c989c00f2708a698f9ca1f"
Nov 02 08:13:42.780762 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:42.780725    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"]
Nov 02 08:13:44.991843 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:13:44.991797    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:13:46.320009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:46.319979    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"
Nov 02 08:13:47.328637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:47.328605    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"
Nov 02 08:13:47.783519 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:47.783472    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl" podUID="3d117963-b74e-4be4-885e-56cb6a205310" containerName="payload-processing" containerID="cri-o://b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7" gracePeriod=25
Nov 02 08:13:48.090911 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:13:48.090776    4898 logging.go:55] [core] [Channel #9322 SubChannel #9323]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.54:50052", ServerName: "10.132.1.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.54:50052: connect: connection refused"
Nov 02 08:13:48.403794 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.403761    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:13:48.438611 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.438573    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-routing-overlay-volume\") pod \"3d117963-b74e-4be4-885e-56cb6a205310\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") "
Nov 02 08:13:48.438812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.438731    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-provider-credentials\") pod \"3d117963-b74e-4be4-885e-56cb6a205310\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") "
Nov 02 08:13:48.438812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.438775    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-kube-api-access-kx8hc\" (UniqueName: \"kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-kube-api-access-kx8hc\") pod \"3d117963-b74e-4be4-885e-56cb6a205310\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") "
Nov 02 08:13:48.438982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.438857    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-plugins-config-volume\") pod \"3d117963-b74e-4be4-885e-56cb6a205310\" (UID: \"3d117963-b74e-4be4-885e-56cb6a205310\") "
Nov 02 08:13:48.439072 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.439043    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-routing-overlay-volume" pod "3d117963-b74e-4be4-885e-56cb6a205310" (UID: "3d117963-b74e-4be4-885e-56cb6a205310"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:48.439313 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.439284    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-plugins-config-volume" pod "3d117963-b74e-4be4-885e-56cb6a205310" (UID: "3d117963-b74e-4be4-885e-56cb6a205310"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:13:48.439627 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.439605    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:48.439706 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.439635    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d117963-b74e-4be4-885e-56cb6a205310-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:48.441250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.441227    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-provider-credentials" pod "3d117963-b74e-4be4-885e-56cb6a205310" (UID: "3d117963-b74e-4be4-885e-56cb6a205310"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:48.441365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.441261    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-kube-api-access-kx8hc" pod "3d117963-b74e-4be4-885e-56cb6a205310" (UID: "3d117963-b74e-4be4-885e-56cb6a205310"). InnerVolumeSpecName "kube-api-access-kx8hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:13:48.510004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.509966    4898 generic.go:356] "Generic (PLEG): container finished" podID="3d117963-b74e-4be4-885e-56cb6a205310" containerID="b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7" exitCode=0
Nov 02 08:13:48.510263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.510065    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl" event={"ID":"3d117963-b74e-4be4-885e-56cb6a205310","Type":"ContainerDied","Data":"b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7"}
Nov 02 08:13:48.510353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.510147    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"
Nov 02 08:13:48.510353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.510310    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl" event={"ID":"3d117963-b74e-4be4-885e-56cb6a205310","Type":"ContainerDied","Data":"e563188363454ba8dbf84499172fdabdfd3c2ec3df58171781f422eef9f15a9a"}
Nov 02 08:13:48.510465 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.510351    4898 scope.go:122] "RemoveContainer" containerID="b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7"
Nov 02 08:13:48.574971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.572304    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:48.574971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.572343    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx8hc\" (UniqueName: \"kubernetes.io/projected/3d117963-b74e-4be4-885e-56cb6a205310-kube-api-access-kx8hc\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:13:48.723630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.723577    4898 scope.go:122] "RemoveContainer" containerID="b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7"
Nov 02 08:13:48.724583 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:13:48.724539    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7\": container with ID starting with b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7 not found: ID does not exist" containerID="b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7"
Nov 02 08:13:48.724798 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.724756    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7"} err="failed to get container status \"b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7\": rpc error: code = NotFound desc = could not find container \"b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7\": container with ID starting with b55b7dbff12454a137ff4c069b3f512f1a4d74dc9b3a7291a58468e9eab175c7 not found: ID does not exist"
Nov 02 08:13:48.763987 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.762471    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"]
Nov 02 08:13:48.772642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:48.772602    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl"]
Nov 02 08:13:49.090486 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:49.090371    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-bb54phvrl" podUID="3d117963-b74e-4be4-885e-56cb6a205310" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.54:50052\" within 1s: context deadline exceeded"
Nov 02 08:13:49.514773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:49.514719    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d117963-b74e-4be4-885e-56cb6a205310" path="/var/lib/kubelet/pods/3d117963-b74e-4be4-885e-56cb6a205310/volumes"
Nov 02 08:13:50.208057 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:50.208015    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"]
Nov 02 08:13:50.220215 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:50.220180    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021a6ddc-4jmlm"]
Nov 02 08:13:51.468619 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:51.468523    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803fbfc1-d231-44be-b5bc-2c021dfd1940" path="/var/lib/kubelet/pods/803fbfc1-d231-44be-b5bc-2c021dfd1940/volumes"
Nov 02 08:13:54.549514 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.549473    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"]
Nov 02 08:13:54.550398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.550378    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3d117963-b74e-4be4-885e-56cb6a205310" containerName="payload-processing"
Nov 02 08:13:54.550459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.550399    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d117963-b74e-4be4-885e-56cb6a205310" containerName="payload-processing"
Nov 02 08:13:54.550459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.550417    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1b408bd6-3dbf-4045-8ce5-b0e1e9abd268" containerName="revoke-keys"
Nov 02 08:13:54.550459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.550423    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b408bd6-3dbf-4045-8ce5-b0e1e9abd268" containerName="revoke-keys"
Nov 02 08:13:54.550588 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.550578    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d117963-b74e-4be4-885e-56cb6a205310" containerName="payload-processing"
Nov 02 08:13:54.550628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.550592    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b408bd6-3dbf-4045-8ce5-b0e1e9abd268" containerName="revoke-keys"
Nov 02 08:13:54.556143 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.556117    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.573753 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.573718    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"]
Nov 02 08:13:54.580929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.579228    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2365fa\"/\"routing-overlay\""
Nov 02 08:13:54.580929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.579326    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2365fa\"/\"agc-cred-d3d9e80e56a5\""
Nov 02 08:13:54.580929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.579341    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2365fa\"/\"kube-root-ca.crt\""
Nov 02 08:13:54.580929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.579395    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2365fa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2365fa\""
Nov 02 08:13:54.580929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.579486    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2365fa\"/\"openshift-service-ca.crt\""
Nov 02 08:13:54.583243 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.583222    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2365fa\"/\"payload-processing-external-model-e2e--d97430f5-dockercfg-fg8tg\""
Nov 02 08:13:54.599782 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.599742    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjglh\" (UniqueName: \"kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-kube-api-access-gjglh\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.600015 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.599801    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.600131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.600050    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.600246 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.600221    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.702110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.702055    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.702523 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.702499    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.704000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.703134    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjglh\" (UniqueName: \"kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-kube-api-access-gjglh\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.704000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.703200    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.710039 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.710002    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.718038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.717914    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.737356 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.737318    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.780632 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.780598    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjglh\" (UniqueName: \"kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-kube-api-access-gjglh\") pod \"payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") " pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:54.878029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:54.877453    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:55.219621 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:13:55.219587    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3f3133_b3e6_4f2d_95b9_e22d311eb5b9.slice/crio-41776ff1877d575fe804f5c1e10d9ed58ca452c8ed2b670141ce0907991188e1 WatchSource:0}: Error finding container 41776ff1877d575fe804f5c1e10d9ed58ca452c8ed2b670141ce0907991188e1: Status 404 returned error can't find the container with id 41776ff1877d575fe804f5c1e10d9ed58ca452c8ed2b670141ce0907991188e1
Nov 02 08:13:55.220867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.220838    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"]
Nov 02 08:13:55.618330 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.618227    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq" event={"ID":"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9","Type":"ContainerStarted","Data":"183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b"}
Nov 02 08:13:55.618330 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.618297    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq" event={"ID":"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9","Type":"ContainerStarted","Data":"41776ff1877d575fe804f5c1e10d9ed58ca452c8ed2b670141ce0907991188e1"}
Nov 02 08:13:55.619018 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.618987    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:13:55.659617 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.659584    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"]
Nov 02 08:13:55.665183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.665156    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.671190 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.671142    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq" podStartSLOduration=1.6711300420000001 podStartE2EDuration="1.671130042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:55.659084781 +0000 UTC m=+2986.349411772" watchObservedRunningTime="2026-11-02 08:13:55.671130042 +0000 UTC m=+2986.361457006"
Nov 02 08:13:55.680397 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.680367    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0f2e06\""
Nov 02 08:13:55.680570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.680370    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"agc-cred-ee21077ba281\""
Nov 02 08:13:55.680656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.680382    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"kube-root-ca.crt\""
Nov 02 08:13:55.680656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.680400    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"openshift-service-ca.crt\""
Nov 02 08:13:55.680656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.680431    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"payload-processing-external-model-e2e--9b11fa18-dockercfg-s7p9c\""
Nov 02 08:13:55.680656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.680437    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0f2e06\"/\"routing-overlay\""
Nov 02 08:13:55.684448 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.684422    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"]
Nov 02 08:13:55.741979 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.741940    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4zg5\" (UniqueName: \"kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-kube-api-access-x4zg5\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.742179 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.742010    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.742266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.742234    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.742464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.742443    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.865910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.847267    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4zg5\" (UniqueName: \"kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-kube-api-access-x4zg5\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.865910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.847348    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.865910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.847478    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.865910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.847636    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.865910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.848763    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.895968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.867157    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.895968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.884756    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.907214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.907138    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4zg5\" (UniqueName: \"kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-kube-api-access-x4zg5\") pod \"payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") " pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:55.985474 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:55.985174    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:56.383597 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:13:56.383559    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab305b0a_1020_4b1c_b3d1_56cc9257bd06.slice/crio-a3e4b4d6c8ecece49cdf0c47cbdfe7812c4bbcf4e1448a10c0429934cf71a54f WatchSource:0}: Error finding container a3e4b4d6c8ecece49cdf0c47cbdfe7812c4bbcf4e1448a10c0429934cf71a54f: Status 404 returned error can't find the container with id a3e4b4d6c8ecece49cdf0c47cbdfe7812c4bbcf4e1448a10c0429934cf71a54f
Nov 02 08:13:56.385999 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:56.385216    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"]
Nov 02 08:13:56.630868 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:56.630759    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq" event={"ID":"ab305b0a-1020-4b1c-b3d1-56cc9257bd06","Type":"ContainerStarted","Data":"12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9"}
Nov 02 08:13:56.630868 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:56.630832    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq" event={"ID":"ab305b0a-1020-4b1c-b3d1-56cc9257bd06","Type":"ContainerStarted","Data":"a3e4b4d6c8ecece49cdf0c47cbdfe7812c4bbcf4e1448a10c0429934cf71a54f"}
Nov 02 08:13:56.630868 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:56.630860    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:13:56.679016 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:56.678951    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq" podStartSLOduration=1.678934646 podStartE2EDuration="1.678934646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:13:56.673682802 +0000 UTC m=+2987.364009769" watchObservedRunningTime="2026-11-02 08:13:56.678934646 +0000 UTC m=+2987.369261611"
Nov 02 08:13:59.283791 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:13:59.283746    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"]
Nov 02 08:14:00.926020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:00.925974    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"]
Nov 02 08:14:00.929916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:00.929860    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:14:02.034634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:02.034589    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"]
Nov 02 08:14:02.046537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:02.046503    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d52e3ac-nzvqm"]
Nov 02 08:14:03.470523 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:03.470478    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e954a21-92cb-45a1-ba7f-e3f079ad73be" path="/var/lib/kubelet/pods/5e954a21-92cb-45a1-ba7f-e3f079ad73be/volumes"
Nov 02 08:14:04.284229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.284173    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq" podUID="ab305b0a-1020-4b1c-b3d1-56cc9257bd06" containerName="payload-processing" containerID="cri-o://12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9" gracePeriod=25
Nov 02 08:14:04.601232 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.601203    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:14:04.725783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.725744    4898 generic.go:356] "Generic (PLEG): container finished" podID="ab305b0a-1020-4b1c-b3d1-56cc9257bd06" containerID="12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9" exitCode=0
Nov 02 08:14:04.725995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.725810    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"
Nov 02 08:14:04.725995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.725828    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq" event={"ID":"ab305b0a-1020-4b1c-b3d1-56cc9257bd06","Type":"ContainerDied","Data":"12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9"}
Nov 02 08:14:04.725995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.725944    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq" event={"ID":"ab305b0a-1020-4b1c-b3d1-56cc9257bd06","Type":"ContainerDied","Data":"a3e4b4d6c8ecece49cdf0c47cbdfe7812c4bbcf4e1448a10c0429934cf71a54f"}
Nov 02 08:14:04.725995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.725978    4898 scope.go:122] "RemoveContainer" containerID="12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9"
Nov 02 08:14:04.738104 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.738078    4898 scope.go:122] "RemoveContainer" containerID="12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9"
Nov 02 08:14:04.738413 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:14:04.738395    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9\": container with ID starting with 12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9 not found: ID does not exist" containerID="12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9"
Nov 02 08:14:04.738455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.738424    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9"} err="failed to get container status \"12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9\": rpc error: code = NotFound desc = could not find container \"12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9\": container with ID starting with 12e987fd89a80af931c1d071b7ef3cfb9c478123ebe2a4d1f2b5bc4430384dd9 not found: ID does not exist"
Nov 02 08:14:04.754289 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.754263    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-provider-credentials\") pod \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") "
Nov 02 08:14:04.754459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.754320    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-kube-api-access-x4zg5\" (UniqueName: \"kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-kube-api-access-x4zg5\") pod \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") "
Nov 02 08:14:04.754514 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.754504    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-routing-overlay-volume\") pod \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") "
Nov 02 08:14:04.754562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.754529    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-plugins-config-volume\") pod \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\" (UID: \"ab305b0a-1020-4b1c-b3d1-56cc9257bd06\") "
Nov 02 08:14:04.754837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.754804    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-routing-overlay-volume" pod "ab305b0a-1020-4b1c-b3d1-56cc9257bd06" (UID: "ab305b0a-1020-4b1c-b3d1-56cc9257bd06"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:14:04.755008 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.754880    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-plugins-config-volume" pod "ab305b0a-1020-4b1c-b3d1-56cc9257bd06" (UID: "ab305b0a-1020-4b1c-b3d1-56cc9257bd06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:14:04.755110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.755083    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:04.755110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.755113    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:04.757129 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.757101    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-kube-api-access-x4zg5" pod "ab305b0a-1020-4b1c-b3d1-56cc9257bd06" (UID: "ab305b0a-1020-4b1c-b3d1-56cc9257bd06"). InnerVolumeSpecName "kube-api-access-x4zg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:14:04.757243 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.757140    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-provider-credentials" pod "ab305b0a-1020-4b1c-b3d1-56cc9257bd06" (UID: "ab305b0a-1020-4b1c-b3d1-56cc9257bd06"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:14:04.857109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.857017    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:04.857109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:04.857054    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4zg5\" (UniqueName: \"kubernetes.io/projected/ab305b0a-1020-4b1c-b3d1-56cc9257bd06-kube-api-access-x4zg5\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:05.119447 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:05.119347    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"]
Nov 02 08:14:05.127353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:05.127319    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0f2e06/payload-processing-external-model-e2e-extmod-a-0f2e06-86f5dp8gq"]
Nov 02 08:14:05.495742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:05.495703    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab305b0a-1020-4b1c-b3d1-56cc9257bd06" path="/var/lib/kubelet/pods/ab305b0a-1020-4b1c-b3d1-56cc9257bd06/volumes"
Nov 02 08:14:05.926531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:05.926480    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq" podUID="de3f3133-b3e6-4f2d-95b9-e22d311eb5b9" containerName="payload-processing" containerID="cri-o://183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b" gracePeriod=25
Nov 02 08:14:06.216188 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.216142    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:14:06.256669 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.256634    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"]
Nov 02 08:14:06.257636 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.257608    4898 cpu_manager.go:414] "containerMap: removing container" podUID="de3f3133-b3e6-4f2d-95b9-e22d311eb5b9" containerName="payload-processing"
Nov 02 08:14:06.257636 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.257634    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3f3133-b3e6-4f2d-95b9-e22d311eb5b9" containerName="payload-processing"
Nov 02 08:14:06.257841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.257653    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ab305b0a-1020-4b1c-b3d1-56cc9257bd06" containerName="payload-processing"
Nov 02 08:14:06.257841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.257661    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab305b0a-1020-4b1c-b3d1-56cc9257bd06" containerName="payload-processing"
Nov 02 08:14:06.257991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.257972    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3f3133-b3e6-4f2d-95b9-e22d311eb5b9" containerName="payload-processing"
Nov 02 08:14:06.258053 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.257996    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab305b0a-1020-4b1c-b3d1-56cc9257bd06" containerName="payload-processing"
Nov 02 08:14:06.263485 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.262228    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.275684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.274105    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"]
Nov 02 08:14:06.299994 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.299947    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2895ef-71fb-4097-829d-bfb60039769f-istiod-ca-cert\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.300189 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.300006    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-socket\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.300189 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.300073    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-istio-token\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.300189 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.300166    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-envoy\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.300321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.300201    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-certs\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.300321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.300234    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2895ef-71fb-4097-829d-bfb60039769f-istio-podinfo\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.300321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.300282    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-data\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.300437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.300352    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhr9n\" (UniqueName: \"kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-kube-api-access-fhr9n\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.300437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.300410    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-credential-socket\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.314547 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.314516    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c4489d-openshift-default-dockercfg-zqk8x\""
Nov 02 08:14:06.401077 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.401035    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-plugins-config-volume\") pod \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") "
Nov 02 08:14:06.401290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.401154    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-routing-overlay-volume\") pod \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") "
Nov 02 08:14:06.401363 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.401343    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-provider-credentials\") pod \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") "
Nov 02 08:14:06.401437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.401422    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-kube-api-access-gjglh\" (UniqueName: \"kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-kube-api-access-gjglh\") pod \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\" (UID: \"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9\") "
Nov 02 08:14:06.401570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.401542    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-plugins-config-volume" pod "de3f3133-b3e6-4f2d-95b9-e22d311eb5b9" (UID: "de3f3133-b3e6-4f2d-95b9-e22d311eb5b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:14:06.401646 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.401582    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-routing-overlay-volume" pod "de3f3133-b3e6-4f2d-95b9-e22d311eb5b9" (UID: "de3f3133-b3e6-4f2d-95b9-e22d311eb5b9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:14:06.401844 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.401825    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2895ef-71fb-4097-829d-bfb60039769f-istiod-ca-cert\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.401915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.401858    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-socket\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.401994 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.401937    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-istio-token\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.402053 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.402038    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-envoy\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.402111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.402074    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-certs\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.402169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.402111    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2895ef-71fb-4097-829d-bfb60039769f-istio-podinfo\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.402233 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.402167    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-data\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.402294 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.402255    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhr9n\" (UniqueName: \"kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-kube-api-access-fhr9n\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.402351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.402322    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-credential-socket\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.402925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.402486    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:06.402925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.402509    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:06.402925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.402706    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2895ef-71fb-4097-829d-bfb60039769f-istiod-ca-cert\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.402925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.402725    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-socket\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.403344 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.403316    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-data\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.403344 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.403335    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-credential-socket\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.403591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.403559    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-certs\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.405054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.405022    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-provider-credentials" pod "de3f3133-b3e6-4f2d-95b9-e22d311eb5b9" (UID: "de3f3133-b3e6-4f2d-95b9-e22d311eb5b9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:14:06.405194 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.405063    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-kube-api-access-gjglh" pod "de3f3133-b3e6-4f2d-95b9-e22d311eb5b9" (UID: "de3f3133-b3e6-4f2d-95b9-e22d311eb5b9"). InnerVolumeSpecName "kube-api-access-gjglh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:14:06.405392 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.405369    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2895ef-71fb-4097-829d-bfb60039769f-istio-podinfo\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.405570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.405548    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-envoy\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.466978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.466645    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhr9n\" (UniqueName: \"kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-kube-api-access-fhr9n\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.500914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.496038    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-istio-token\") pod \"e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") " pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.516909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.511835    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:06.516909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.511872    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjglh\" (UniqueName: \"kubernetes.io/projected/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9-kube-api-access-gjglh\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:06.595909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.594998    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:06.756679 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.756274    4898 generic.go:356] "Generic (PLEG): container finished" podID="de3f3133-b3e6-4f2d-95b9-e22d311eb5b9" containerID="183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b" exitCode=0
Nov 02 08:14:06.756679 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.756473    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq" event={"ID":"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9","Type":"ContainerDied","Data":"183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b"}
Nov 02 08:14:06.756679 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.756519    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq" event={"ID":"de3f3133-b3e6-4f2d-95b9-e22d311eb5b9","Type":"ContainerDied","Data":"41776ff1877d575fe804f5c1e10d9ed58ca452c8ed2b670141ce0907991188e1"}
Nov 02 08:14:06.756679 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.756544    4898 scope.go:122] "RemoveContainer" containerID="183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b"
Nov 02 08:14:06.756679 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.756654    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"
Nov 02 08:14:06.788247 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.787780    4898 scope.go:122] "RemoveContainer" containerID="183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b"
Nov 02 08:14:06.788600 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:14:06.788493    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b\": container with ID starting with 183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b not found: ID does not exist" containerID="183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b"
Nov 02 08:14:06.788600 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.788528    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b"} err="failed to get container status \"183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b\": rpc error: code = NotFound desc = could not find container \"183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b\": container with ID starting with 183077a191fb4d226aa74a82d005f06a07503918a6dbf02e626075a8e15c820b not found: ID does not exist"
Nov 02 08:14:06.820547 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.820511    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"]
Nov 02 08:14:06.836175 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.836124    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2365fa/payload-processing-external-model-e2e-extmod-b-2365fa-8578mx9kq"]
Nov 02 08:14:06.868958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:06.865320    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"]
Nov 02 08:14:07.092001 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:14:07.088928    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2895ef_71fb_4097_829d_bfb60039769f.slice/crio-ab49b12642b4ea1d666428a1d2df354bed0f523e6740a2f38e80268d95a87475 WatchSource:0}: Error finding container ab49b12642b4ea1d666428a1d2df354bed0f523e6740a2f38e80268d95a87475: Status 404 returned error can't find the container with id ab49b12642b4ea1d666428a1d2df354bed0f523e6740a2f38e80268d95a87475
Nov 02 08:14:07.092001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:07.091612    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"]
Nov 02 08:14:07.092778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:07.092532    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:14:07.092778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:07.092625    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:14:07.092778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:07.092674    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:14:07.470417 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:07.470375    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de3f3133-b3e6-4f2d-95b9-e22d311eb5b9" path="/var/lib/kubelet/pods/de3f3133-b3e6-4f2d-95b9-e22d311eb5b9/volumes"
Nov 02 08:14:07.772034 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:07.771933    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" event={"ID":"0d2895ef-71fb-4097-829d-bfb60039769f","Type":"ContainerStarted","Data":"6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116"}
Nov 02 08:14:07.772034 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:07.772032    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" event={"ID":"0d2895ef-71fb-4097-829d-bfb60039769f","Type":"ContainerStarted","Data":"ab49b12642b4ea1d666428a1d2df354bed0f523e6740a2f38e80268d95a87475"}
Nov 02 08:14:07.828973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:07.828881    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" podStartSLOduration=1.828867072 podStartE2EDuration="1.828867072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:14:07.82825095 +0000 UTC m=+2998.518577913" watchObservedRunningTime="2026-11-02 08:14:07.828867072 +0000 UTC m=+2998.519194036"
Nov 02 08:14:08.596061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:08.596014    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:08.603880 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:08.603845    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:08.783213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:08.783178    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:08.784520 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:08.784491    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:14:08.911345 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:08.911305    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"]
Nov 02 08:14:10.491269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:10.491234    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:14:10.514120 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:10.514007    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:14:11.866677 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:11.866599    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9" podUID="e6f4a436-b299-4503-9c1f-cd8ef0baff9e" containerName="payload-processing" containerID="cri-o://ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869" gracePeriod=25
Nov 02 08:14:12.228593 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.223907    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"]
Nov 02 08:14:12.228593 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.224039    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:14:12.228593 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.224854    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e6f4a436-b299-4503-9c1f-cd8ef0baff9e" containerName="payload-processing"
Nov 02 08:14:12.228593 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.224870    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f4a436-b299-4503-9c1f-cd8ef0baff9e" containerName="payload-processing"
Nov 02 08:14:12.228593 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.225054    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f4a436-b299-4503-9c1f-cd8ef0baff9e" containerName="payload-processing"
Nov 02 08:14:12.261795 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.253237    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"
Nov 02 08:14:12.295938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.285504    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c4489d-dockercfg-h9zxg\""
Nov 02 08:14:12.314006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.304335    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c4489d\""
Nov 02 08:14:12.351957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.348020    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-provider-credentials\") pod \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") "
Nov 02 08:14:12.351957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.348174    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-plugins-config-volume\") pod \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") "
Nov 02 08:14:12.352919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.352383    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-routing-overlay-volume\") pod \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") "
Nov 02 08:14:12.357537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.357488    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-routing-overlay-volume" pod "e6f4a436-b299-4503-9c1f-cd8ef0baff9e" (UID: "e6f4a436-b299-4503-9c1f-cd8ef0baff9e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:14:12.358103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.357062    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-kube-api-access-rqs82\" (UniqueName: \"kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-kube-api-access-rqs82\") pod \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\" (UID: \"e6f4a436-b299-4503-9c1f-cd8ef0baff9e\") "
Nov 02 08:14:12.374949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.358195    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-plugins-config-volume" pod "e6f4a436-b299-4503-9c1f-cd8ef0baff9e" (UID: "e6f4a436-b299-4503-9c1f-cd8ef0baff9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:14:12.375931 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.375846    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqhqp\" (UniqueName: \"kubernetes.io/projected/c32626e9-75be-4a2b-b16b-3c5242be4989-kube-api-access-qqhqp\") pod \"payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42\" (UID: \"c32626e9-75be-4a2b-b16b-3c5242be4989\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"
Nov 02 08:14:12.376726 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.376698    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32626e9-75be-4a2b-b16b-3c5242be4989-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42\" (UID: \"c32626e9-75be-4a2b-b16b-3c5242be4989\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"
Nov 02 08:14:12.386814 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.377121    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:12.386814 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.377152    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:12.401008 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.387996    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-kube-api-access-rqs82" pod "e6f4a436-b299-4503-9c1f-cd8ef0baff9e" (UID: "e6f4a436-b299-4503-9c1f-cd8ef0baff9e"). InnerVolumeSpecName "kube-api-access-rqs82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:14:12.401008 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.394070    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-provider-credentials" pod "e6f4a436-b299-4503-9c1f-cd8ef0baff9e" (UID: "e6f4a436-b299-4503-9c1f-cd8ef0baff9e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:14:12.401341 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.401309    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"]
Nov 02 08:14:12.579978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.573685    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32626e9-75be-4a2b-b16b-3c5242be4989-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42\" (UID: \"c32626e9-75be-4a2b-b16b-3c5242be4989\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"
Nov 02 08:14:12.580558 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.580521    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqhqp\" (UniqueName: \"kubernetes.io/projected/c32626e9-75be-4a2b-b16b-3c5242be4989-kube-api-access-qqhqp\") pod \"payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42\" (UID: \"c32626e9-75be-4a2b-b16b-3c5242be4989\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"
Nov 02 08:14:12.587921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.581027    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqs82\" (UniqueName: \"kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-kube-api-access-rqs82\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:12.605851 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.605815    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6f4a436-b299-4503-9c1f-cd8ef0baff9e-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:12.606538 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.606510    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32626e9-75be-4a2b-b16b-3c5242be4989-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42\" (UID: \"c32626e9-75be-4a2b-b16b-3c5242be4989\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"
Nov 02 08:14:12.626072 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.626036    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqhqp\" (UniqueName: \"kubernetes.io/projected/c32626e9-75be-4a2b-b16b-3c5242be4989-kube-api-access-qqhqp\") pod \"payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42\" (UID: \"c32626e9-75be-4a2b-b16b-3c5242be4989\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"
Nov 02 08:14:12.824004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.823959    4898 generic.go:356] "Generic (PLEG): container finished" podID="e6f4a436-b299-4503-9c1f-cd8ef0baff9e" containerID="ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869" exitCode=0
Nov 02 08:14:12.824177 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.824011    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"
Nov 02 08:14:12.824177 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.824025    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9" event={"ID":"e6f4a436-b299-4503-9c1f-cd8ef0baff9e","Type":"ContainerDied","Data":"ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869"}
Nov 02 08:14:12.824177 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.824086    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9" event={"ID":"e6f4a436-b299-4503-9c1f-cd8ef0baff9e","Type":"ContainerDied","Data":"5c123c285110f95bcf06d0d8727c92d3e9f1d11086d22e93abe45af21ce3d280"}
Nov 02 08:14:12.824177 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.824117    4898 scope.go:122] "RemoveContainer" containerID="ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869"
Nov 02 08:14:12.838533 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.838506    4898 scope.go:122] "RemoveContainer" containerID="ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869"
Nov 02 08:14:12.838922 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:14:12.838880    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869\": container with ID starting with ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869 not found: ID does not exist" containerID="ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869"
Nov 02 08:14:12.839008 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.838931    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869"} err="failed to get container status \"ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869\": rpc error: code = NotFound desc = could not find container \"ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869\": container with ID starting with ad4ccde185ee08c395f0c33a9547eaaa8ab1c385f53e85204a0583982db9b869 not found: ID does not exist"
Nov 02 08:14:12.878808 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.878773    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"]
Nov 02 08:14:12.885542 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.885506    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf48ct9"]
Nov 02 08:14:12.902683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:12.902645    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"
Nov 02 08:14:13.407913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:13.401569    4898 scope.go:122] "RemoveContainer" containerID="046fd2d33455f38350a9b5f9f14cc8741f76f9b8fe414a99ac1eed3311a9ff39"
Nov 02 08:14:13.530055 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:13.530019    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f4a436-b299-4503-9c1f-cd8ef0baff9e" path="/var/lib/kubelet/pods/e6f4a436-b299-4503-9c1f-cd8ef0baff9e/volumes"
Nov 02 08:14:13.532113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:13.532085    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"]
Nov 02 08:14:13.548016 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:13.547979    4898 scope.go:122] "RemoveContainer" containerID="ff11e009f60dfed34a84f15d60fcdc2bfc55c243ba8e2a99b3c1e0d0188e22cc"
Nov 02 08:14:13.592462 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:13.592178    4898 scope.go:122] "RemoveContainer" containerID="782b7f0ca6bf087f7ef8f4852dc37455398297ef9f10be7ca0e828ff2a39bf97"
Nov 02 08:14:13.835577 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:13.835537    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42" event={"ID":"c32626e9-75be-4a2b-b16b-3c5242be4989","Type":"ContainerStarted","Data":"b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4"}
Nov 02 08:14:13.835711 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:13.835640    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"
Nov 02 08:14:13.835711 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:13.835664    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42" event={"ID":"c32626e9-75be-4a2b-b16b-3c5242be4989","Type":"ContainerStarted","Data":"83010389436eff78bd1cfeba888beaa18eb8d1584c794ed514484dc8c9ea5d88"}
Nov 02 08:14:13.889637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:13.880748    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42" podStartSLOduration=1.880730306 podStartE2EDuration="1.880730306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:14:13.879341105 +0000 UTC m=+3004.569668072" watchObservedRunningTime="2026-11-02 08:14:13.880730306 +0000 UTC m=+3004.571057271"
Nov 02 08:14:13.912035 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:13.911959    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn" podUID="49a6eabb-e243-437a-9eb2-f048ea5ab124" containerName="payload-processing" containerID="cri-o://4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75" gracePeriod=25
Nov 02 08:14:14.033964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.033927    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"]
Nov 02 08:14:14.040757 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.040723    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-270f250e-687qg"]
Nov 02 08:14:14.280096 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.280065    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:14:14.479911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.476625    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-provider-credentials\" (UniqueName: \"kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-provider-credentials\") pod \"49a6eabb-e243-437a-9eb2-f048ea5ab124\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") "
Nov 02 08:14:14.479911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.476715    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-routing-overlay-volume\") pod \"49a6eabb-e243-437a-9eb2-f048ea5ab124\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") "
Nov 02 08:14:14.479911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.476859    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-plugins-config-volume\") pod \"49a6eabb-e243-437a-9eb2-f048ea5ab124\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") "
Nov 02 08:14:14.479911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.477057    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-kube-api-access-958q4\" (UniqueName: \"kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-kube-api-access-958q4\") pod \"49a6eabb-e243-437a-9eb2-f048ea5ab124\" (UID: \"49a6eabb-e243-437a-9eb2-f048ea5ab124\") "
Nov 02 08:14:14.497616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.482464    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-plugins-config-volume" pod "49a6eabb-e243-437a-9eb2-f048ea5ab124" (UID: "49a6eabb-e243-437a-9eb2-f048ea5ab124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:14:14.497616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.483088    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-routing-overlay-volume" pod "49a6eabb-e243-437a-9eb2-f048ea5ab124" (UID: "49a6eabb-e243-437a-9eb2-f048ea5ab124"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:14:14.497616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.483218    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-kube-api-access-958q4" pod "49a6eabb-e243-437a-9eb2-f048ea5ab124" (UID: "49a6eabb-e243-437a-9eb2-f048ea5ab124"). InnerVolumeSpecName "kube-api-access-958q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:14:14.497616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.497308    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-provider-credentials" pod "49a6eabb-e243-437a-9eb2-f048ea5ab124" (UID: "49a6eabb-e243-437a-9eb2-f048ea5ab124"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:14:14.579042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.578995    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-958q4\" (UniqueName: \"kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-kube-api-access-958q4\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:14.579042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.579031    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49a6eabb-e243-437a-9eb2-f048ea5ab124-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:14.579042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.579040    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:14.579042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.579049    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6eabb-e243-437a-9eb2-f048ea5ab124-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:14:14.849838 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.849739    4898 generic.go:356] "Generic (PLEG): container finished" podID="49a6eabb-e243-437a-9eb2-f048ea5ab124" containerID="4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75" exitCode=0
Nov 02 08:14:14.849838 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.849812    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"
Nov 02 08:14:14.850091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.849828    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn" event={"ID":"49a6eabb-e243-437a-9eb2-f048ea5ab124","Type":"ContainerDied","Data":"4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75"}
Nov 02 08:14:14.850091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.849935    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn" event={"ID":"49a6eabb-e243-437a-9eb2-f048ea5ab124","Type":"ContainerDied","Data":"d2c0e0a2e4f7a47695dd1204a8b4e660025ffa227476b370d0f42b95913a1db4"}
Nov 02 08:14:14.850091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.849972    4898 scope.go:122] "RemoveContainer" containerID="4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75"
Nov 02 08:14:14.863789 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.863762    4898 scope.go:122] "RemoveContainer" containerID="4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75"
Nov 02 08:14:14.864184 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:14:14.864153    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75\": container with ID starting with 4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75 not found: ID does not exist" containerID="4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75"
Nov 02 08:14:14.864300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.864193    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75"} err="failed to get container status \"4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75\": rpc error: code = NotFound desc = could not find container \"4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75\": container with ID starting with 4c6221ed2dc3cb8e4183656ff181af37c087cb685b1e1bfda422d09f6f120e75 not found: ID does not exist"
Nov 02 08:14:14.944676 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.944642    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"]
Nov 02 08:14:14.956366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:14.956335    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf74ggrn"]
Nov 02 08:14:14.990722 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:14:14.990684    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:14:15.469869 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:15.469832    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a6eabb-e243-437a-9eb2-f048ea5ab124" path="/var/lib/kubelet/pods/49a6eabb-e243-437a-9eb2-f048ea5ab124/volumes"
Nov 02 08:14:15.470454 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:15.470434    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b82c790c-ea55-4cea-bceb-ccd4eac21567" path="/var/lib/kubelet/pods/b82c790c-ea55-4cea-bceb-ccd4eac21567/volumes"
Nov 02 08:14:24.034183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:24.034092    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"]
Nov 02 08:14:24.043661 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:24.043632    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b222260-q25xl"]
Nov 02 08:14:24.854053 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:24.854021    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"
Nov 02 08:14:25.467934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:25.467881    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72054409-d8c7-4201-ad57-4eb8a67ded7a" path="/var/lib/kubelet/pods/72054409-d8c7-4201-ad57-4eb8a67ded7a/volumes"
Nov 02 08:14:35.031706 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:35.031657    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"]
Nov 02 08:14:35.043798 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:35.043761    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c95e5285-sl8dn"]
Nov 02 08:14:35.468350 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:35.468308    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741b8f34-3bac-4607-a2dc-e97073dd9582" path="/var/lib/kubelet/pods/741b8f34-3bac-4607-a2dc-e97073dd9582/volumes"
Nov 02 08:14:38.032626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:38.032591    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"]
Nov 02 08:14:38.044375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:38.044345    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6097cd29-px5qz"]
Nov 02 08:14:39.468001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:14:39.467964    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de8f11aa-420b-4c67-aeb7-2eb5a62cebb5" path="/var/lib/kubelet/pods/de8f11aa-420b-4c67-aeb7-2eb5a62cebb5/volumes"
Nov 02 08:14:44.990258 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:14:44.990217    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:15:00.146375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.146331    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"]
Nov 02 08:15:00.147418 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.147393    4898 cpu_manager.go:414] "containerMap: removing container" podUID="49a6eabb-e243-437a-9eb2-f048ea5ab124" containerName="payload-processing"
Nov 02 08:15:00.147418 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.147419    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a6eabb-e243-437a-9eb2-f048ea5ab124" containerName="payload-processing"
Nov 02 08:15:00.147636 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.147618    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a6eabb-e243-437a-9eb2-f048ea5ab124" containerName="payload-processing"
Nov 02 08:15:00.151791 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.151770    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"]
Nov 02 08:15:00.151963 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.151925    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"
Nov 02 08:15:00.181168 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.181129    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:15:00.181367 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.181239    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:15:00.181367 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.181334    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:15:00.209910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.209185    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"]
Nov 02 08:15:00.209910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.209480    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"
Nov 02 08:15:00.216766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.216091    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"]
Nov 02 08:15:00.217567 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.217256    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"
Nov 02 08:15:00.223095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.222774    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"]
Nov 02 08:15:00.223095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.223068    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"
Nov 02 08:15:00.229328 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.228877    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"
Nov 02 08:15:00.229328 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.229281    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"]
Nov 02 08:15:00.237905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.237866    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"]
Nov 02 08:15:00.238242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.238220    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"
Nov 02 08:15:00.276214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.276183    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"]
Nov 02 08:15:00.276562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.276544    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"]
Nov 02 08:15:00.276753 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.276740    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"]
Nov 02 08:15:00.276992 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.276964    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"
Nov 02 08:15:00.309488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.294005    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"]
Nov 02 08:15:00.309488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.306082    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"]
Nov 02 08:15:00.316098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.316062    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbh87\" (UniqueName: \"kubernetes.io/projected/7ebd6238-df9e-4a53-8f56-66b32a3167fd-kube-api-access-kbh87\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7\" (UID: \"7ebd6238-df9e-4a53-8f56-66b32a3167fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"
Nov 02 08:15:00.318102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.318070    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebd6238-df9e-4a53-8f56-66b32a3167fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7\" (UID: \"7ebd6238-df9e-4a53-8f56-66b32a3167fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"
Nov 02 08:15:00.330715 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.330683    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"]
Nov 02 08:15:00.331027 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.330950    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"
Nov 02 08:15:00.336747 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.336723    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"]
Nov 02 08:15:00.336873 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.336786    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"]
Nov 02 08:15:00.336965 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.336923    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"
Nov 02 08:15:00.340761 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.340733    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"]
Nov 02 08:15:00.350184 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.350159    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"
Nov 02 08:15:00.361846 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.361817    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"]
Nov 02 08:15:00.367750 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.367721    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"]
Nov 02 08:15:00.382817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.382795    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"]
Nov 02 08:15:00.383043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.383018    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"
Nov 02 08:15:00.383332 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.383317    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"]
Nov 02 08:15:00.389065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.389042    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"
Nov 02 08:15:00.389969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.389883    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"]
Nov 02 08:15:00.397116 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.397059    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"]
Nov 02 08:15:00.397227 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.397156    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"
Nov 02 08:15:00.399172 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.399149    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"]
Nov 02 08:15:00.403556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.403535    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"]
Nov 02 08:15:00.407631 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.407611    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"]
Nov 02 08:15:00.412807 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.412789    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"]
Nov 02 08:15:00.412939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.412817    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"
Nov 02 08:15:00.417227 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.417206    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"]
Nov 02 08:15:00.417338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.417299    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"
Nov 02 08:15:00.419533 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.419496    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdds4\" (UniqueName: \"kubernetes.io/projected/c81cf60f-13a6-4260-9364-114e89edeef7-kube-api-access-hdds4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw\" (UID: \"c81cf60f-13a6-4260-9364-114e89edeef7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"
Nov 02 08:15:00.419646 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.419553    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff46f33-1e5a-4a41-85a1-7d18107f3844-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz\" (UID: \"5ff46f33-1e5a-4a41-85a1-7d18107f3844\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"
Nov 02 08:15:00.419713 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.419694    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6\" (UID: \"9584c1c3-65cf-4e75-aa0f-9a3681010cc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"
Nov 02 08:15:00.419774 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.419719    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ef0f94-811c-4542-aac5-f4ba52c18f27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd\" (UID: \"13ef0f94-811c-4542-aac5-f4ba52c18f27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"
Nov 02 08:15:00.419834 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.419781    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf2pb\" (UniqueName: \"kubernetes.io/projected/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-kube-api-access-hf2pb\") pod \"maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s\" (UID: \"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"
Nov 02 08:15:00.419834 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.419810    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95edc7f-4b4b-449d-870c-156648685af2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv\" (UID: \"d95edc7f-4b4b-449d-870c-156648685af2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"
Nov 02 08:15:00.419974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.419866    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlklk\" (UniqueName: \"kubernetes.io/projected/d95edc7f-4b4b-449d-870c-156648685af2-kube-api-access-jlklk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv\" (UID: \"d95edc7f-4b4b-449d-870c-156648685af2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"
Nov 02 08:15:00.419974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.419917    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5wws\" (UniqueName: \"kubernetes.io/projected/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-kube-api-access-b5wws\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz\" (UID: \"a6cfc1ff-3ea1-4845-98c2-d02d836507a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"
Nov 02 08:15:00.419974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.419940    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6rq6\" (UniqueName: \"kubernetes.io/projected/a2876d3b-6372-4dd1-919d-4a53af9540bb-kube-api-access-k6rq6\") pod \"maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l\" (UID: \"a2876d3b-6372-4dd1-919d-4a53af9540bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"
Nov 02 08:15:00.420146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420053    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f648095-5449-42df-885a-29e8044cb8c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv\" (UID: \"1f648095-5449-42df-885a-29e8044cb8c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"
Nov 02 08:15:00.420146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420074    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvgm7\" (UniqueName: \"kubernetes.io/projected/ee823ad5-4625-4083-9cc9-20044043dedf-kube-api-access-qvgm7\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88\" (UID: \"ee823ad5-4625-4083-9cc9-20044043dedf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"
Nov 02 08:15:00.420146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420098    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab5290b-4dd7-494c-9381-a467e78f96c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq\" (UID: \"fab5290b-4dd7-494c-9381-a467e78f96c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"
Nov 02 08:15:00.420338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420191    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nglnw\" (UniqueName: \"kubernetes.io/projected/7bca6afb-be58-48e6-8d4f-317c45edcffd-kube-api-access-nglnw\") pod \"maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8\" (UID: \"7bca6afb-be58-48e6-8d4f-317c45edcffd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"
Nov 02 08:15:00.420338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420222    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbh87\" (UniqueName: \"kubernetes.io/projected/7ebd6238-df9e-4a53-8f56-66b32a3167fd-kube-api-access-kbh87\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7\" (UID: \"7ebd6238-df9e-4a53-8f56-66b32a3167fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"
Nov 02 08:15:00.420338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420247    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfd9\" (UniqueName: \"kubernetes.io/projected/c3200858-4b8b-4500-bc2f-8fa185f752f6-kube-api-access-bnfd9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq\" (UID: \"c3200858-4b8b-4500-bc2f-8fa185f752f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"
Nov 02 08:15:00.420338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420283    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9dxc\" (UniqueName: \"kubernetes.io/projected/13ef0f94-811c-4542-aac5-f4ba52c18f27-kube-api-access-c9dxc\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd\" (UID: \"13ef0f94-811c-4542-aac5-f4ba52c18f27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"
Nov 02 08:15:00.420338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420306    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qpw\" (UniqueName: \"kubernetes.io/projected/fab5290b-4dd7-494c-9381-a467e78f96c8-kube-api-access-z7qpw\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq\" (UID: \"fab5290b-4dd7-494c-9381-a467e78f96c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"
Nov 02 08:15:00.420338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420325    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s\" (UID: \"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"
Nov 02 08:15:00.420686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420357    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bca6afb-be58-48e6-8d4f-317c45edcffd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8\" (UID: \"7bca6afb-be58-48e6-8d4f-317c45edcffd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"
Nov 02 08:15:00.420686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420391    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81cf60f-13a6-4260-9364-114e89edeef7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw\" (UID: \"c81cf60f-13a6-4260-9364-114e89edeef7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"
Nov 02 08:15:00.420686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420408    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee823ad5-4625-4083-9cc9-20044043dedf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88\" (UID: \"ee823ad5-4625-4083-9cc9-20044043dedf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"
Nov 02 08:15:00.420686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420424    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvh4\" (UniqueName: \"kubernetes.io/projected/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-kube-api-access-vqvh4\") pod \"maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6\" (UID: \"9584c1c3-65cf-4e75-aa0f-9a3681010cc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"
Nov 02 08:15:00.420686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420450    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebd6238-df9e-4a53-8f56-66b32a3167fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7\" (UID: \"7ebd6238-df9e-4a53-8f56-66b32a3167fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"
Nov 02 08:15:00.420686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420479    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz\" (UID: \"a6cfc1ff-3ea1-4845-98c2-d02d836507a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"
Nov 02 08:15:00.420686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420534    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84784\" (UniqueName: \"kubernetes.io/projected/5ff46f33-1e5a-4a41-85a1-7d18107f3844-kube-api-access-84784\") pod \"maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz\" (UID: \"5ff46f33-1e5a-4a41-85a1-7d18107f3844\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"
Nov 02 08:15:00.420686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420601    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gvjc\" (UniqueName: \"kubernetes.io/projected/1f648095-5449-42df-885a-29e8044cb8c2-kube-api-access-7gvjc\") pod \"maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv\" (UID: \"1f648095-5449-42df-885a-29e8044cb8c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"
Nov 02 08:15:00.420686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420648    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3200858-4b8b-4500-bc2f-8fa185f752f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq\" (UID: \"c3200858-4b8b-4500-bc2f-8fa185f752f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"
Nov 02 08:15:00.420686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420666    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2876d3b-6372-4dd1-919d-4a53af9540bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l\" (UID: \"a2876d3b-6372-4dd1-919d-4a53af9540bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"
Nov 02 08:15:00.421274 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.420974    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebd6238-df9e-4a53-8f56-66b32a3167fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7\" (UID: \"7ebd6238-df9e-4a53-8f56-66b32a3167fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"
Nov 02 08:15:00.422357 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.422338    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"]
Nov 02 08:15:00.422461 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.422413    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"
Nov 02 08:15:00.430426 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.430403    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"]
Nov 02 08:15:00.448492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.445579    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"]
Nov 02 08:15:00.448492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.445812    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbh87\" (UniqueName: \"kubernetes.io/projected/7ebd6238-df9e-4a53-8f56-66b32a3167fd-kube-api-access-kbh87\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7\" (UID: \"7ebd6238-df9e-4a53-8f56-66b32a3167fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"
Nov 02 08:15:00.456292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.456258    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"]
Nov 02 08:15:00.456292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.456280    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"
Nov 02 08:15:00.461177 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.461150    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"]
Nov 02 08:15:00.469264 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.468909    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"]
Nov 02 08:15:00.483337 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.483311    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"]
Nov 02 08:15:00.484742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.484722    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"
Nov 02 08:15:00.521631 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.521594    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3200858-4b8b-4500-bc2f-8fa185f752f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq\" (UID: \"c3200858-4b8b-4500-bc2f-8fa185f752f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"
Nov 02 08:15:00.521854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.521645    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2876d3b-6372-4dd1-919d-4a53af9540bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l\" (UID: \"a2876d3b-6372-4dd1-919d-4a53af9540bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"
Nov 02 08:15:00.521854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.521689    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdds4\" (UniqueName: \"kubernetes.io/projected/c81cf60f-13a6-4260-9364-114e89edeef7-kube-api-access-hdds4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw\" (UID: \"c81cf60f-13a6-4260-9364-114e89edeef7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"
Nov 02 08:15:00.521854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.521742    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff46f33-1e5a-4a41-85a1-7d18107f3844-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz\" (UID: \"5ff46f33-1e5a-4a41-85a1-7d18107f3844\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"
Nov 02 08:15:00.521854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.521800    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzjws\" (UniqueName: \"kubernetes.io/projected/e2da54f0-7885-4cd9-969c-c6051ba28f41-kube-api-access-lzjws\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48\" (UID: \"e2da54f0-7885-4cd9-969c-c6051ba28f41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"
Nov 02 08:15:00.521854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.521829    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da54f0-7885-4cd9-969c-c6051ba28f41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48\" (UID: \"e2da54f0-7885-4cd9-969c-c6051ba28f41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.521882    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6\" (UID: \"9584c1c3-65cf-4e75-aa0f-9a3681010cc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.521927    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ef0f94-811c-4542-aac5-f4ba52c18f27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd\" (UID: \"13ef0f94-811c-4542-aac5-f4ba52c18f27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.521951    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blw8w\" (UniqueName: \"kubernetes.io/projected/d8186095-8ca2-42ab-aedf-d0d44811258a-kube-api-access-blw8w\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc\" (UID: \"d8186095-8ca2-42ab-aedf-d0d44811258a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.522124    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf2pb\" (UniqueName: \"kubernetes.io/projected/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-kube-api-access-hf2pb\") pod \"maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s\" (UID: \"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.522194    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mldts\" (UniqueName: \"kubernetes.io/projected/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-kube-api-access-mldts\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff\" (UID: \"0615ef0f-a1d8-47cc-a6b4-481fadc306bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.522238    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95edc7f-4b4b-449d-870c-156648685af2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv\" (UID: \"d95edc7f-4b4b-449d-870c-156648685af2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.522268    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlklk\" (UniqueName: \"kubernetes.io/projected/d95edc7f-4b4b-449d-870c-156648685af2-kube-api-access-jlklk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv\" (UID: \"d95edc7f-4b4b-449d-870c-156648685af2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.522391    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5wws\" (UniqueName: \"kubernetes.io/projected/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-kube-api-access-b5wws\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz\" (UID: \"a6cfc1ff-3ea1-4845-98c2-d02d836507a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.522468    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rq6\" (UniqueName: \"kubernetes.io/projected/a2876d3b-6372-4dd1-919d-4a53af9540bb-kube-api-access-k6rq6\") pod \"maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l\" (UID: \"a2876d3b-6372-4dd1-919d-4a53af9540bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.522470    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3200858-4b8b-4500-bc2f-8fa185f752f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq\" (UID: \"c3200858-4b8b-4500-bc2f-8fa185f752f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523194    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6\" (UID: \"9584c1c3-65cf-4e75-aa0f-9a3681010cc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523313    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff46f33-1e5a-4a41-85a1-7d18107f3844-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz\" (UID: \"5ff46f33-1e5a-4a41-85a1-7d18107f3844\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523401    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f648095-5449-42df-885a-29e8044cb8c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv\" (UID: \"1f648095-5449-42df-885a-29e8044cb8c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523453    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvgm7\" (UniqueName: \"kubernetes.io/projected/ee823ad5-4625-4083-9cc9-20044043dedf-kube-api-access-qvgm7\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88\" (UID: \"ee823ad5-4625-4083-9cc9-20044043dedf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523499    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab5290b-4dd7-494c-9381-a467e78f96c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq\" (UID: \"fab5290b-4dd7-494c-9381-a467e78f96c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523594    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nglnw\" (UniqueName: \"kubernetes.io/projected/7bca6afb-be58-48e6-8d4f-317c45edcffd-kube-api-access-nglnw\") pod \"maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8\" (UID: \"7bca6afb-be58-48e6-8d4f-317c45edcffd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523662    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfd9\" (UniqueName: \"kubernetes.io/projected/c3200858-4b8b-4500-bc2f-8fa185f752f6-kube-api-access-bnfd9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq\" (UID: \"c3200858-4b8b-4500-bc2f-8fa185f752f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523695    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dxc\" (UniqueName: \"kubernetes.io/projected/13ef0f94-811c-4542-aac5-f4ba52c18f27-kube-api-access-c9dxc\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd\" (UID: \"13ef0f94-811c-4542-aac5-f4ba52c18f27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523727    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qpw\" (UniqueName: \"kubernetes.io/projected/fab5290b-4dd7-494c-9381-a467e78f96c8-kube-api-access-z7qpw\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq\" (UID: \"fab5290b-4dd7-494c-9381-a467e78f96c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523757    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s\" (UID: \"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523790    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bca6afb-be58-48e6-8d4f-317c45edcffd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8\" (UID: \"7bca6afb-be58-48e6-8d4f-317c45edcffd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523856    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81cf60f-13a6-4260-9364-114e89edeef7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw\" (UID: \"c81cf60f-13a6-4260-9364-114e89edeef7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523907    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee823ad5-4625-4083-9cc9-20044043dedf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88\" (UID: \"ee823ad5-4625-4083-9cc9-20044043dedf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523922    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ef0f94-811c-4542-aac5-f4ba52c18f27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd\" (UID: \"13ef0f94-811c-4542-aac5-f4ba52c18f27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523934    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvh4\" (UniqueName: \"kubernetes.io/projected/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-kube-api-access-vqvh4\") pod \"maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6\" (UID: \"9584c1c3-65cf-4e75-aa0f-9a3681010cc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523878    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95edc7f-4b4b-449d-870c-156648685af2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv\" (UID: \"d95edc7f-4b4b-449d-870c-156648685af2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.523988    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz\" (UID: \"a6cfc1ff-3ea1-4845-98c2-d02d836507a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.524024    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8186095-8ca2-42ab-aedf-d0d44811258a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc\" (UID: \"d8186095-8ca2-42ab-aedf-d0d44811258a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.524055    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84784\" (UniqueName: \"kubernetes.io/projected/5ff46f33-1e5a-4a41-85a1-7d18107f3844-kube-api-access-84784\") pod \"maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz\" (UID: \"5ff46f33-1e5a-4a41-85a1-7d18107f3844\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.524093    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gvjc\" (UniqueName: \"kubernetes.io/projected/1f648095-5449-42df-885a-29e8044cb8c2-kube-api-access-7gvjc\") pod \"maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv\" (UID: \"1f648095-5449-42df-885a-29e8044cb8c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.524108    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f648095-5449-42df-885a-29e8044cb8c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv\" (UID: \"1f648095-5449-42df-885a-29e8044cb8c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"
Nov 02 08:15:00.524182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.524149    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff\" (UID: \"0615ef0f-a1d8-47cc-a6b4-481fadc306bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"
Nov 02 08:15:00.526730 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.525397    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81cf60f-13a6-4260-9364-114e89edeef7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw\" (UID: \"c81cf60f-13a6-4260-9364-114e89edeef7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"
Nov 02 08:15:00.526730 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.525531    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab5290b-4dd7-494c-9381-a467e78f96c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq\" (UID: \"fab5290b-4dd7-494c-9381-a467e78f96c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"
Nov 02 08:15:00.526730 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.526003    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz\" (UID: \"a6cfc1ff-3ea1-4845-98c2-d02d836507a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"
Nov 02 08:15:00.526730 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.526116    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee823ad5-4625-4083-9cc9-20044043dedf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88\" (UID: \"ee823ad5-4625-4083-9cc9-20044043dedf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"
Nov 02 08:15:00.533004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.532979    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bca6afb-be58-48e6-8d4f-317c45edcffd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8\" (UID: \"7bca6afb-be58-48e6-8d4f-317c45edcffd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"
Nov 02 08:15:00.533121 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.533009    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2876d3b-6372-4dd1-919d-4a53af9540bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l\" (UID: \"a2876d3b-6372-4dd1-919d-4a53af9540bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"
Nov 02 08:15:00.533268 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.533249    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s\" (UID: \"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"
Nov 02 08:15:00.546712 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.545972    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5wws\" (UniqueName: \"kubernetes.io/projected/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-kube-api-access-b5wws\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz\" (UID: \"a6cfc1ff-3ea1-4845-98c2-d02d836507a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"
Nov 02 08:15:00.546834 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.546767    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf2pb\" (UniqueName: \"kubernetes.io/projected/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-kube-api-access-hf2pb\") pod \"maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s\" (UID: \"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"
Nov 02 08:15:00.547365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.547330    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdds4\" (UniqueName: \"kubernetes.io/projected/c81cf60f-13a6-4260-9364-114e89edeef7-kube-api-access-hdds4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw\" (UID: \"c81cf60f-13a6-4260-9364-114e89edeef7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"
Nov 02 08:15:00.548022 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.548000    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rq6\" (UniqueName: \"kubernetes.io/projected/a2876d3b-6372-4dd1-919d-4a53af9540bb-kube-api-access-k6rq6\") pod \"maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l\" (UID: \"a2876d3b-6372-4dd1-919d-4a53af9540bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"
Nov 02 08:15:00.551597 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.551563    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9dxc\" (UniqueName: \"kubernetes.io/projected/13ef0f94-811c-4542-aac5-f4ba52c18f27-kube-api-access-c9dxc\") pod \"maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd\" (UID: \"13ef0f94-811c-4542-aac5-f4ba52c18f27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"
Nov 02 08:15:00.552249 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.552221    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlklk\" (UniqueName: \"kubernetes.io/projected/d95edc7f-4b4b-449d-870c-156648685af2-kube-api-access-jlklk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv\" (UID: \"d95edc7f-4b4b-449d-870c-156648685af2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"
Nov 02 08:15:00.552817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.552269    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvgm7\" (UniqueName: \"kubernetes.io/projected/ee823ad5-4625-4083-9cc9-20044043dedf-kube-api-access-qvgm7\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88\" (UID: \"ee823ad5-4625-4083-9cc9-20044043dedf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"
Nov 02 08:15:00.552987 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.552761    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qpw\" (UniqueName: \"kubernetes.io/projected/fab5290b-4dd7-494c-9381-a467e78f96c8-kube-api-access-z7qpw\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq\" (UID: \"fab5290b-4dd7-494c-9381-a467e78f96c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"
Nov 02 08:15:00.553056 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.553005    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfd9\" (UniqueName: \"kubernetes.io/projected/c3200858-4b8b-4500-bc2f-8fa185f752f6-kube-api-access-bnfd9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq\" (UID: \"c3200858-4b8b-4500-bc2f-8fa185f752f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"
Nov 02 08:15:00.563603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.562794    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84784\" (UniqueName: \"kubernetes.io/projected/5ff46f33-1e5a-4a41-85a1-7d18107f3844-kube-api-access-84784\") pod \"maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz\" (UID: \"5ff46f33-1e5a-4a41-85a1-7d18107f3844\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"
Nov 02 08:15:00.563603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.563391    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"
Nov 02 08:15:00.569925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.568447    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nglnw\" (UniqueName: \"kubernetes.io/projected/7bca6afb-be58-48e6-8d4f-317c45edcffd-kube-api-access-nglnw\") pod \"maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8\" (UID: \"7bca6afb-be58-48e6-8d4f-317c45edcffd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"
Nov 02 08:15:00.569925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.568630    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gvjc\" (UniqueName: \"kubernetes.io/projected/1f648095-5449-42df-885a-29e8044cb8c2-kube-api-access-7gvjc\") pod \"maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv\" (UID: \"1f648095-5449-42df-885a-29e8044cb8c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"
Nov 02 08:15:00.571355 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.571331    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvh4\" (UniqueName: \"kubernetes.io/projected/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-kube-api-access-vqvh4\") pod \"maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6\" (UID: \"9584c1c3-65cf-4e75-aa0f-9a3681010cc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"
Nov 02 08:15:00.588504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.588466    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"
Nov 02 08:15:00.627466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.625195    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"
Nov 02 08:15:00.630577 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.629638    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8186095-8ca2-42ab-aedf-d0d44811258a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc\" (UID: \"d8186095-8ca2-42ab-aedf-d0d44811258a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"
Nov 02 08:15:00.630577 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.629716    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff\" (UID: \"0615ef0f-a1d8-47cc-a6b4-481fadc306bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"
Nov 02 08:15:00.630577 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.629831    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzjws\" (UniqueName: \"kubernetes.io/projected/e2da54f0-7885-4cd9-969c-c6051ba28f41-kube-api-access-lzjws\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48\" (UID: \"e2da54f0-7885-4cd9-969c-c6051ba28f41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"
Nov 02 08:15:00.630577 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.629862    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da54f0-7885-4cd9-969c-c6051ba28f41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48\" (UID: \"e2da54f0-7885-4cd9-969c-c6051ba28f41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"
Nov 02 08:15:00.630577 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.629969    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blw8w\" (UniqueName: \"kubernetes.io/projected/d8186095-8ca2-42ab-aedf-d0d44811258a-kube-api-access-blw8w\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc\" (UID: \"d8186095-8ca2-42ab-aedf-d0d44811258a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"
Nov 02 08:15:00.630577 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.630019    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mldts\" (UniqueName: \"kubernetes.io/projected/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-kube-api-access-mldts\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff\" (UID: \"0615ef0f-a1d8-47cc-a6b4-481fadc306bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"
Nov 02 08:15:00.631341 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.631306    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8186095-8ca2-42ab-aedf-d0d44811258a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc\" (UID: \"d8186095-8ca2-42ab-aedf-d0d44811258a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"
Nov 02 08:15:00.631968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.631919    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff\" (UID: \"0615ef0f-a1d8-47cc-a6b4-481fadc306bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"
Nov 02 08:15:00.633465 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.633437    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da54f0-7885-4cd9-969c-c6051ba28f41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48\" (UID: \"e2da54f0-7885-4cd9-969c-c6051ba28f41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"
Nov 02 08:15:00.642705 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.642630    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"
Nov 02 08:15:00.657734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.657686    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"
Nov 02 08:15:00.675705 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.675671    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"
Nov 02 08:15:00.683990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.683923    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mldts\" (UniqueName: \"kubernetes.io/projected/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-kube-api-access-mldts\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff\" (UID: \"0615ef0f-a1d8-47cc-a6b4-481fadc306bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"
Nov 02 08:15:00.686247 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.686170    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"
Nov 02 08:15:00.693366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.690089    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzjws\" (UniqueName: \"kubernetes.io/projected/e2da54f0-7885-4cd9-969c-c6051ba28f41-kube-api-access-lzjws\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48\" (UID: \"e2da54f0-7885-4cd9-969c-c6051ba28f41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"
Nov 02 08:15:00.693366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.690674    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blw8w\" (UniqueName: \"kubernetes.io/projected/d8186095-8ca2-42ab-aedf-d0d44811258a-kube-api-access-blw8w\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc\" (UID: \"d8186095-8ca2-42ab-aedf-d0d44811258a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"
Nov 02 08:15:00.702203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.702119    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"
Nov 02 08:15:00.713698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.713501    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"
Nov 02 08:15:00.719796 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.719611    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"
Nov 02 08:15:00.736421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.734122    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"
Nov 02 08:15:00.746684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.746470    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"
Nov 02 08:15:00.828918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.825265    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"
Nov 02 08:15:00.828918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.828687    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"
Nov 02 08:15:00.828918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.828792    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"
Nov 02 08:15:00.846158 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.846067    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"
Nov 02 08:15:00.951050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:00.951015    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"]
Nov 02 08:15:01.004539 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:01.004491    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebd6238_df9e_4a53_8f56_66b32a3167fd.slice/crio-07a2d598a097b7f0474a61242860e17fd127edd86f0c46b19ae8f436ffec6881 WatchSource:0}: Error finding container 07a2d598a097b7f0474a61242860e17fd127edd86f0c46b19ae8f436ffec6881: Status 404 returned error can't find the container with id 07a2d598a097b7f0474a61242860e17fd127edd86f0c46b19ae8f436ffec6881
Nov 02 08:15:01.175227 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.171392    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"]
Nov 02 08:15:01.228722 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.228688    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"]
Nov 02 08:15:01.407011 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.406487    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7" event={"ID":"7ebd6238-df9e-4a53-8f56-66b32a3167fd","Type":"ContainerStarted","Data":"07a2d598a097b7f0474a61242860e17fd127edd86f0c46b19ae8f436ffec6881"}
Nov 02 08:15:01.408216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.408187    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq" event={"ID":"fab5290b-4dd7-494c-9381-a467e78f96c8","Type":"ContainerStarted","Data":"c3071f78cdd272038646ee4d90b6f7a90824b7e95788927e64a6f86f4dc207a6"}
Nov 02 08:15:01.409611 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.409589    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv" event={"ID":"1f648095-5449-42df-885a-29e8044cb8c2","Type":"ContainerStarted","Data":"e8a620bf46e603198967438d337177feea8228c917b1cd67ff5f28b39ec2a5ed"}
Nov 02 08:15:01.502840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.502808    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"]
Nov 02 08:15:01.516357 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.513094    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"]
Nov 02 08:15:01.576395 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:01.575486    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff46f33_1e5a_4a41_85a1_7d18107f3844.slice/crio-305e921db7870511f08920ea9c08bad04ad11dc27975656daf5a938b7b082bb4 WatchSource:0}: Error finding container 305e921db7870511f08920ea9c08bad04ad11dc27975656daf5a938b7b082bb4: Status 404 returned error can't find the container with id 305e921db7870511f08920ea9c08bad04ad11dc27975656daf5a938b7b082bb4
Nov 02 08:15:01.579211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.579188    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"]
Nov 02 08:15:01.617124 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.617006    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"]
Nov 02 08:15:01.627267 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.627240    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"]
Nov 02 08:15:01.749280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.749251    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"]
Nov 02 08:15:01.767121 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.767092    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"]
Nov 02 08:15:01.774481 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:01.774446    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95edc7f_4b4b_449d_870c_156648685af2.slice/crio-c95c1407b399908eecbc2fb096c7b506dfc2171da08a8f0fadaa891c53f4b29b WatchSource:0}: Error finding container c95c1407b399908eecbc2fb096c7b506dfc2171da08a8f0fadaa891c53f4b29b: Status 404 returned error can't find the container with id c95c1407b399908eecbc2fb096c7b506dfc2171da08a8f0fadaa891c53f4b29b
Nov 02 08:15:01.778981 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.778946    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"]
Nov 02 08:15:01.806254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.806216    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"]
Nov 02 08:15:01.832829 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.832798    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"]
Nov 02 08:15:01.845940 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:01.845908    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf3abb8_f4bd_4770_8ce7_5df5defd7fb5.slice/crio-f2dbb8adb7a63f5ba4ce1b33ad8537a880d7cb1a73d5fb04475b0617f65759c4 WatchSource:0}: Error finding container f2dbb8adb7a63f5ba4ce1b33ad8537a880d7cb1a73d5fb04475b0617f65759c4: Status 404 returned error can't find the container with id f2dbb8adb7a63f5ba4ce1b33ad8537a880d7cb1a73d5fb04475b0617f65759c4
Nov 02 08:15:01.869748 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.869661    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"]
Nov 02 08:15:01.875252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.875219    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"]
Nov 02 08:15:01.899218 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.899154    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"]
Nov 02 08:15:01.904255 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:01.904231    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"]
Nov 02 08:15:02.160500 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:02.160446    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0615ef0f_a1d8_47cc_a6b4_481fadc306bc.slice/crio-3d2c76d0ed7433d64cfba36db644ba9ee4fa045b18fed7f6f45d803c9cc2bb60 WatchSource:0}: Error finding container 3d2c76d0ed7433d64cfba36db644ba9ee4fa045b18fed7f6f45d803c9cc2bb60: Status 404 returned error can't find the container with id 3d2c76d0ed7433d64cfba36db644ba9ee4fa045b18fed7f6f45d803c9cc2bb60
Nov 02 08:15:02.165831 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:02.165794    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ef0f94_811c_4542_aac5_f4ba52c18f27.slice/crio-2141a15d0192e84a2eff63bd509d1bc801ca94d5aaa584c68847f24280d11d05 WatchSource:0}: Error finding container 2141a15d0192e84a2eff63bd509d1bc801ca94d5aaa584c68847f24280d11d05: Status 404 returned error can't find the container with id 2141a15d0192e84a2eff63bd509d1bc801ca94d5aaa584c68847f24280d11d05
Nov 02 08:15:02.261654 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:02.261618    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8186095_8ca2_42ab_aedf_d0d44811258a.slice/crio-7d74125d1d5f8e3937be14665dd802cc64aab150eca67de056e98fb04a5e5706 WatchSource:0}: Error finding container 7d74125d1d5f8e3937be14665dd802cc64aab150eca67de056e98fb04a5e5706: Status 404 returned error can't find the container with id 7d74125d1d5f8e3937be14665dd802cc64aab150eca67de056e98fb04a5e5706
Nov 02 08:15:02.426952 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.426871    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv" event={"ID":"d95edc7f-4b4b-449d-870c-156648685af2","Type":"ContainerStarted","Data":"c95c1407b399908eecbc2fb096c7b506dfc2171da08a8f0fadaa891c53f4b29b"}
Nov 02 08:15:02.428532 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.428505    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc" event={"ID":"d8186095-8ca2-42ab-aedf-d0d44811258a","Type":"ContainerStarted","Data":"7d74125d1d5f8e3937be14665dd802cc64aab150eca67de056e98fb04a5e5706"}
Nov 02 08:15:02.430481 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.430450    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq" event={"ID":"c3200858-4b8b-4500-bc2f-8fa185f752f6","Type":"ContainerStarted","Data":"bcec8ffcf60e191504f8bca6c6a7d6b685ce9e11c4763d424034d9f2a54a8840"}
Nov 02 08:15:02.430582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.430506    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq" event={"ID":"c3200858-4b8b-4500-bc2f-8fa185f752f6","Type":"ContainerStarted","Data":"5f63face04ec92c8d4fa96bd7b8297569b12a7f4f4fd0e2700a24d7db75e6e4f"}
Nov 02 08:15:02.434067 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.434035    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l" event={"ID":"a2876d3b-6372-4dd1-919d-4a53af9540bb","Type":"ContainerStarted","Data":"fb6e368de6fdd8eccab236788c0e2e246ae090a07809b7c561d167c5ce29c6a8"}
Nov 02 08:15:02.434181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.434090    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l" event={"ID":"a2876d3b-6372-4dd1-919d-4a53af9540bb","Type":"ContainerStarted","Data":"3a61de2f327b362aeb0549c72e194296224fb641b49cf35ca5c3c36ea62c0413"}
Nov 02 08:15:02.477152 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.472319    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8" event={"ID":"7bca6afb-be58-48e6-8d4f-317c45edcffd","Type":"ContainerStarted","Data":"af3e69eee79f3cf15ea9b39a65b0e4ffee06488ef3a103bbee878b0b0aecc770"}
Nov 02 08:15:02.477152 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.472392    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8" event={"ID":"7bca6afb-be58-48e6-8d4f-317c45edcffd","Type":"ContainerStarted","Data":"0c26d0173478a031a58a8b8b45c6d7cd578c396891eda10291a024dfa73c5295"}
Nov 02 08:15:02.481501 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.481280    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq" podStartSLOduration=2.481261372 podStartE2EDuration="2.481261372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.47986962 +0000 UTC m=+3053.170196595" watchObservedRunningTime="2026-11-02 08:15:02.481261372 +0000 UTC m=+3053.171588339"
Nov 02 08:15:02.487783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.487752    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7" event={"ID":"7ebd6238-df9e-4a53-8f56-66b32a3167fd","Type":"ContainerStarted","Data":"c79171882ad8e7c54b9506a696083134c863d0dce72238acd335a35c7b056bb0"}
Nov 02 08:15:02.501397 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.501297    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48" event={"ID":"e2da54f0-7885-4cd9-969c-c6051ba28f41","Type":"ContainerStarted","Data":"b97ae07ee35983f66ac1f2df056400b4203a5039039c66f1fe8c7312d86ef9ee"}
Nov 02 08:15:02.501397 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.501359    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48" event={"ID":"e2da54f0-7885-4cd9-969c-c6051ba28f41","Type":"ContainerStarted","Data":"90bbcc1bf29a4c045b5c7b573e848aed89de897258000d3f4ead9d31274b97ed"}
Nov 02 08:15:02.504667 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.504584    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88" event={"ID":"ee823ad5-4625-4083-9cc9-20044043dedf","Type":"ContainerStarted","Data":"8dcf327bb0f64943788a99097911f2384050676d096592a57b81f7ac76620a05"}
Nov 02 08:15:02.508856 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.508829    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq" event={"ID":"fab5290b-4dd7-494c-9381-a467e78f96c8","Type":"ContainerStarted","Data":"48706592fe209b45294efb6b39084cf20ca330bdbe40c8b1c5f8edc5d54ea77e"}
Nov 02 08:15:02.511801 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.511715    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz" event={"ID":"a6cfc1ff-3ea1-4845-98c2-d02d836507a2","Type":"ContainerStarted","Data":"9366a10d2d4607d1c55c445e7f3b52062c93ef39d815e9cf66995ed6493cb5d0"}
Nov 02 08:15:02.539198 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.539141    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6" event={"ID":"9584c1c3-65cf-4e75-aa0f-9a3681010cc2","Type":"ContainerStarted","Data":"7ed7b8800bca385b0540dddee8fe8298f8d45989fd503442da0356f343d637df"}
Nov 02 08:15:02.539561 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.539479    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6" event={"ID":"9584c1c3-65cf-4e75-aa0f-9a3681010cc2","Type":"ContainerStarted","Data":"05277329815a0d6d4605abdc3005738b3fa06c84c2e5af56e391138ec8e9d3d8"}
Nov 02 08:15:02.541277 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.541244    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd" event={"ID":"13ef0f94-811c-4542-aac5-f4ba52c18f27","Type":"ContainerStarted","Data":"2141a15d0192e84a2eff63bd509d1bc801ca94d5aaa584c68847f24280d11d05"}
Nov 02 08:15:02.543155 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.543122    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff" event={"ID":"0615ef0f-a1d8-47cc-a6b4-481fadc306bc","Type":"ContainerStarted","Data":"3d2c76d0ed7433d64cfba36db644ba9ee4fa045b18fed7f6f45d803c9cc2bb60"}
Nov 02 08:15:02.545212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.545186    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv" event={"ID":"1f648095-5449-42df-885a-29e8044cb8c2","Type":"ContainerStarted","Data":"1fd602f0dc7dbc5493ba5fccabf2d0fb7c9e6e475a75d768362ac5d9971de281"}
Nov 02 08:15:02.548039 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.548012    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz" event={"ID":"5ff46f33-1e5a-4a41-85a1-7d18107f3844","Type":"ContainerStarted","Data":"25d66d094bf9b5cdb9b247166247c23d3a0b7a95b7eb669c884937dedf6ce668"}
Nov 02 08:15:02.548178 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.548058    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz" event={"ID":"5ff46f33-1e5a-4a41-85a1-7d18107f3844","Type":"ContainerStarted","Data":"305e921db7870511f08920ea9c08bad04ad11dc27975656daf5a938b7b082bb4"}
Nov 02 08:15:02.549714 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.549672    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s" event={"ID":"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5","Type":"ContainerStarted","Data":"f2dbb8adb7a63f5ba4ce1b33ad8537a880d7cb1a73d5fb04475b0617f65759c4"}
Nov 02 08:15:02.552482 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.551857    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw" event={"ID":"c81cf60f-13a6-4260-9364-114e89edeef7","Type":"ContainerStarted","Data":"4e901b40c3f01dd7eaf30f909be7fab3eadb84dda7b49dc794103ebb2b7b7852"}
Nov 02 08:15:02.552482 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.551945    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw" event={"ID":"c81cf60f-13a6-4260-9364-114e89edeef7","Type":"ContainerStarted","Data":"c958c7a5dc72ad93efa1729138122cc614450f90c2d87dfa7caef59a22a5ebb0"}
Nov 02 08:15:02.576376 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.576317    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l" podStartSLOduration=2.5762980090000003 podStartE2EDuration="2.576298009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.532616141 +0000 UTC m=+3053.222943108" watchObservedRunningTime="2026-11-02 08:15:02.576298009 +0000 UTC m=+3053.266624973"
Nov 02 08:15:02.579776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.579726    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8" podStartSLOduration=2.579712111 podStartE2EDuration="2.579712111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.572330106 +0000 UTC m=+3053.262657090" watchObservedRunningTime="2026-11-02 08:15:02.579712111 +0000 UTC m=+3053.270039080"
Nov 02 08:15:02.620108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.620060    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz" podStartSLOduration=2.620045928 podStartE2EDuration="2.620045928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.618144757 +0000 UTC m=+3053.308471721" watchObservedRunningTime="2026-11-02 08:15:02.620045928 +0000 UTC m=+3053.310372911"
Nov 02 08:15:02.673128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.673064    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7" podStartSLOduration=2.673044664 podStartE2EDuration="2.673044664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.670652564 +0000 UTC m=+3053.360979547" watchObservedRunningTime="2026-11-02 08:15:02.673044664 +0000 UTC m=+3053.363371630"
Nov 02 08:15:02.719924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.718038    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6" podStartSLOduration=2.718013313 podStartE2EDuration="2.718013313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.711883875 +0000 UTC m=+3053.402210857" watchObservedRunningTime="2026-11-02 08:15:02.718013313 +0000 UTC m=+3053.408340297"
Nov 02 08:15:02.785776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.785698    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw" podStartSLOduration=2.785678607 podStartE2EDuration="2.785678607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.777401951 +0000 UTC m=+3053.467728917" watchObservedRunningTime="2026-11-02 08:15:02.785678607 +0000 UTC m=+3053.476005572"
Nov 02 08:15:02.822293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.822239    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq" podStartSLOduration=2.822225866 podStartE2EDuration="2.822225866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.817538662 +0000 UTC m=+3053.507865622" watchObservedRunningTime="2026-11-02 08:15:02.822225866 +0000 UTC m=+3053.512552830"
Nov 02 08:15:02.861286 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.861229    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv" podStartSLOduration=2.8611963940000003 podStartE2EDuration="2.861196394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.856718916 +0000 UTC m=+3053.547045881" watchObservedRunningTime="2026-11-02 08:15:02.861196394 +0000 UTC m=+3053.551523358"
Nov 02 08:15:02.914260 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:02.914181    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48" podStartSLOduration=2.914159399 podStartE2EDuration="2.914159399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:02.912468185 +0000 UTC m=+3053.602795150" watchObservedRunningTime="2026-11-02 08:15:02.914159399 +0000 UTC m=+3053.604486364"
Nov 02 08:15:03.573260 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.573215    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd" event={"ID":"13ef0f94-811c-4542-aac5-f4ba52c18f27","Type":"ContainerStarted","Data":"3b8588fa7d62954c1426d8b142b8c8f65dd02d7b79416435de0dd4cedb478d9c"}
Nov 02 08:15:03.577241 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.577208    4898 generic.go:356] "Generic (PLEG): container finished" podID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" containerID="84418be6ff68cedf99bdf017e9cdcb9421c72886e5b99eefc37959baae5a8876" exitCode=60
Nov 02 08:15:03.577377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.577280    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff" event={"ID":"0615ef0f-a1d8-47cc-a6b4-481fadc306bc","Type":"ContainerDied","Data":"84418be6ff68cedf99bdf017e9cdcb9421c72886e5b99eefc37959baae5a8876"}
Nov 02 08:15:03.578079 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.578052    4898 scope.go:122] "RemoveContainer" containerID="84418be6ff68cedf99bdf017e9cdcb9421c72886e5b99eefc37959baae5a8876"
Nov 02 08:15:03.580391 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.580364    4898 generic.go:356] "Generic (PLEG): container finished" podID="1f648095-5449-42df-885a-29e8044cb8c2" containerID="1fd602f0dc7dbc5493ba5fccabf2d0fb7c9e6e475a75d768362ac5d9971de281" exitCode=60
Nov 02 08:15:03.580496 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.580442    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv" event={"ID":"1f648095-5449-42df-885a-29e8044cb8c2","Type":"ContainerDied","Data":"1fd602f0dc7dbc5493ba5fccabf2d0fb7c9e6e475a75d768362ac5d9971de281"}
Nov 02 08:15:03.580824 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.580804    4898 scope.go:122] "RemoveContainer" containerID="1fd602f0dc7dbc5493ba5fccabf2d0fb7c9e6e475a75d768362ac5d9971de281"
Nov 02 08:15:03.584556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.584529    4898 generic.go:356] "Generic (PLEG): container finished" podID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" containerID="25d66d094bf9b5cdb9b247166247c23d3a0b7a95b7eb669c884937dedf6ce668" exitCode=60
Nov 02 08:15:03.584670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.584623    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz" event={"ID":"5ff46f33-1e5a-4a41-85a1-7d18107f3844","Type":"ContainerDied","Data":"25d66d094bf9b5cdb9b247166247c23d3a0b7a95b7eb669c884937dedf6ce668"}
Nov 02 08:15:03.584866 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.584846    4898 scope.go:122] "RemoveContainer" containerID="25d66d094bf9b5cdb9b247166247c23d3a0b7a95b7eb669c884937dedf6ce668"
Nov 02 08:15:03.586865 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.586836    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s" event={"ID":"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5","Type":"ContainerStarted","Data":"f26e2d10012cb1202bf7b688a73557207f8993c7065a5ce60be30095f46709b6"}
Nov 02 08:15:03.587333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.587312    4898 scope.go:122] "RemoveContainer" containerID="f26e2d10012cb1202bf7b688a73557207f8993c7065a5ce60be30095f46709b6"
Nov 02 08:15:03.593169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.593123    4898 generic.go:356] "Generic (PLEG): container finished" podID="c81cf60f-13a6-4260-9364-114e89edeef7" containerID="4e901b40c3f01dd7eaf30f909be7fab3eadb84dda7b49dc794103ebb2b7b7852" exitCode=60
Nov 02 08:15:03.593299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.593273    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw" event={"ID":"c81cf60f-13a6-4260-9364-114e89edeef7","Type":"ContainerDied","Data":"4e901b40c3f01dd7eaf30f909be7fab3eadb84dda7b49dc794103ebb2b7b7852"}
Nov 02 08:15:03.593456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.593436    4898 scope.go:122] "RemoveContainer" containerID="4e901b40c3f01dd7eaf30f909be7fab3eadb84dda7b49dc794103ebb2b7b7852"
Nov 02 08:15:03.602082 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.601728    4898 generic.go:356] "Generic (PLEG): container finished" podID="d95edc7f-4b4b-449d-870c-156648685af2" containerID="0f8a68f225d7cd7b6de960a9d7bd374ee92f7815f7eabfc1af654f4f1e399ab4" exitCode=60
Nov 02 08:15:03.602165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.602146    4898 scope.go:122] "RemoveContainer" containerID="0f8a68f225d7cd7b6de960a9d7bd374ee92f7815f7eabfc1af654f4f1e399ab4"
Nov 02 08:15:03.602257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.602236    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv" event={"ID":"d95edc7f-4b4b-449d-870c-156648685af2","Type":"ContainerDied","Data":"0f8a68f225d7cd7b6de960a9d7bd374ee92f7815f7eabfc1af654f4f1e399ab4"}
Nov 02 08:15:03.606778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.606748    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc" event={"ID":"d8186095-8ca2-42ab-aedf-d0d44811258a","Type":"ContainerStarted","Data":"d3a006b072469b48570589117950afaf86eca3f09d182d80cf5257ecf9dbe740"}
Nov 02 08:15:03.615865 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.615017    4898 generic.go:356] "Generic (PLEG): container finished" podID="c3200858-4b8b-4500-bc2f-8fa185f752f6" containerID="bcec8ffcf60e191504f8bca6c6a7d6b685ce9e11c4763d424034d9f2a54a8840" exitCode=60
Nov 02 08:15:03.615865 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.615097    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq" event={"ID":"c3200858-4b8b-4500-bc2f-8fa185f752f6","Type":"ContainerDied","Data":"bcec8ffcf60e191504f8bca6c6a7d6b685ce9e11c4763d424034d9f2a54a8840"}
Nov 02 08:15:03.615865 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.615441    4898 scope.go:122] "RemoveContainer" containerID="bcec8ffcf60e191504f8bca6c6a7d6b685ce9e11c4763d424034d9f2a54a8840"
Nov 02 08:15:03.619333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.619301    4898 generic.go:356] "Generic (PLEG): container finished" podID="a2876d3b-6372-4dd1-919d-4a53af9540bb" containerID="fb6e368de6fdd8eccab236788c0e2e246ae090a07809b7c561d167c5ce29c6a8" exitCode=60
Nov 02 08:15:03.619440 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.619346    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l" event={"ID":"a2876d3b-6372-4dd1-919d-4a53af9540bb","Type":"ContainerDied","Data":"fb6e368de6fdd8eccab236788c0e2e246ae090a07809b7c561d167c5ce29c6a8"}
Nov 02 08:15:03.619645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.619627    4898 scope.go:122] "RemoveContainer" containerID="fb6e368de6fdd8eccab236788c0e2e246ae090a07809b7c561d167c5ce29c6a8"
Nov 02 08:15:03.621560 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.621539    4898 generic.go:356] "Generic (PLEG): container finished" podID="7bca6afb-be58-48e6-8d4f-317c45edcffd" containerID="af3e69eee79f3cf15ea9b39a65b0e4ffee06488ef3a103bbee878b0b0aecc770" exitCode=60
Nov 02 08:15:03.621741 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.621722    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8" event={"ID":"7bca6afb-be58-48e6-8d4f-317c45edcffd","Type":"ContainerDied","Data":"af3e69eee79f3cf15ea9b39a65b0e4ffee06488ef3a103bbee878b0b0aecc770"}
Nov 02 08:15:03.621876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.621820    4898 scope.go:122] "RemoveContainer" containerID="af3e69eee79f3cf15ea9b39a65b0e4ffee06488ef3a103bbee878b0b0aecc770"
Nov 02 08:15:03.622154 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.622110    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd" podStartSLOduration=3.622097537 podStartE2EDuration="3.622097537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:03.617490621 +0000 UTC m=+3054.307817586" watchObservedRunningTime="2026-11-02 08:15:03.622097537 +0000 UTC m=+3054.312424507"
Nov 02 08:15:03.623933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.623872    4898 generic.go:356] "Generic (PLEG): container finished" podID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" containerID="c79171882ad8e7c54b9506a696083134c863d0dce72238acd335a35c7b056bb0" exitCode=60
Nov 02 08:15:03.624068 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.624050    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7" event={"ID":"7ebd6238-df9e-4a53-8f56-66b32a3167fd","Type":"ContainerDied","Data":"c79171882ad8e7c54b9506a696083134c863d0dce72238acd335a35c7b056bb0"}
Nov 02 08:15:03.624401 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.624381    4898 scope.go:122] "RemoveContainer" containerID="c79171882ad8e7c54b9506a696083134c863d0dce72238acd335a35c7b056bb0"
Nov 02 08:15:03.630009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.629986    4898 generic.go:356] "Generic (PLEG): container finished" podID="e2da54f0-7885-4cd9-969c-c6051ba28f41" containerID="b97ae07ee35983f66ac1f2df056400b4203a5039039c66f1fe8c7312d86ef9ee" exitCode=60
Nov 02 08:15:03.630235 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.630068    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48" event={"ID":"e2da54f0-7885-4cd9-969c-c6051ba28f41","Type":"ContainerDied","Data":"b97ae07ee35983f66ac1f2df056400b4203a5039039c66f1fe8c7312d86ef9ee"}
Nov 02 08:15:03.630373 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.630353    4898 scope.go:122] "RemoveContainer" containerID="b97ae07ee35983f66ac1f2df056400b4203a5039039c66f1fe8c7312d86ef9ee"
Nov 02 08:15:03.632449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.632423    4898 generic.go:356] "Generic (PLEG): container finished" podID="ee823ad5-4625-4083-9cc9-20044043dedf" containerID="240d0cb823cc583f76316c68ee3fb756eeae7e74d6c4942a0cd4c4c2395e371c" exitCode=60
Nov 02 08:15:03.632600 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.632527    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88" event={"ID":"ee823ad5-4625-4083-9cc9-20044043dedf","Type":"ContainerDied","Data":"240d0cb823cc583f76316c68ee3fb756eeae7e74d6c4942a0cd4c4c2395e371c"}
Nov 02 08:15:03.633329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.633308    4898 scope.go:122] "RemoveContainer" containerID="240d0cb823cc583f76316c68ee3fb756eeae7e74d6c4942a0cd4c4c2395e371c"
Nov 02 08:15:03.634623 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.634600    4898 generic.go:356] "Generic (PLEG): container finished" podID="fab5290b-4dd7-494c-9381-a467e78f96c8" containerID="48706592fe209b45294efb6b39084cf20ca330bdbe40c8b1c5f8edc5d54ea77e" exitCode=60
Nov 02 08:15:03.634729 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.634680    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq" event={"ID":"fab5290b-4dd7-494c-9381-a467e78f96c8","Type":"ContainerDied","Data":"48706592fe209b45294efb6b39084cf20ca330bdbe40c8b1c5f8edc5d54ea77e"}
Nov 02 08:15:03.635092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.635071    4898 scope.go:122] "RemoveContainer" containerID="48706592fe209b45294efb6b39084cf20ca330bdbe40c8b1c5f8edc5d54ea77e"
Nov 02 08:15:03.638867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.637497    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz" event={"ID":"a6cfc1ff-3ea1-4845-98c2-d02d836507a2","Type":"ContainerStarted","Data":"f76ad9d5e3c338a4ef8d979aaf529eb4962647502e3692747cd2de47be07125e"}
Nov 02 08:15:03.638867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.637751    4898 scope.go:122] "RemoveContainer" containerID="f76ad9d5e3c338a4ef8d979aaf529eb4962647502e3692747cd2de47be07125e"
Nov 02 08:15:03.644141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.644117    4898 generic.go:356] "Generic (PLEG): container finished" podID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" containerID="7ed7b8800bca385b0540dddee8fe8298f8d45989fd503442da0356f343d637df" exitCode=60
Nov 02 08:15:03.644243 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.644213    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6" event={"ID":"9584c1c3-65cf-4e75-aa0f-9a3681010cc2","Type":"ContainerDied","Data":"7ed7b8800bca385b0540dddee8fe8298f8d45989fd503442da0356f343d637df"}
Nov 02 08:15:03.644537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.644511    4898 scope.go:122] "RemoveContainer" containerID="7ed7b8800bca385b0540dddee8fe8298f8d45989fd503442da0356f343d637df"
Nov 02 08:15:03.749669 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:03.749619    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc" podStartSLOduration=3.7496067220000002 podStartE2EDuration="3.749606722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:03.746636468 +0000 UTC m=+3054.436963449" watchObservedRunningTime="2026-11-02 08:15:03.749606722 +0000 UTC m=+3054.439933683"
Nov 02 08:15:04.673000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.672967    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv" event={"ID":"1f648095-5449-42df-885a-29e8044cb8c2","Type":"ContainerStarted","Data":"8fd44eb5808145b94b4867fe641a9f8688b4a9e0ae70903069b5befc3ea90734"}
Nov 02 08:15:04.676322 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.676290    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz" event={"ID":"5ff46f33-1e5a-4a41-85a1-7d18107f3844","Type":"ContainerStarted","Data":"18f9173ea272240ed1bdcfb60555a5e221f08c491d89d449a8290b2fcd44ae06"}
Nov 02 08:15:04.679038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.679006    4898 generic.go:356] "Generic (PLEG): container finished" podID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" containerID="f26e2d10012cb1202bf7b688a73557207f8993c7065a5ce60be30095f46709b6" exitCode=60
Nov 02 08:15:04.679161 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.679102    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s" event={"ID":"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5","Type":"ContainerDied","Data":"f26e2d10012cb1202bf7b688a73557207f8993c7065a5ce60be30095f46709b6"}
Nov 02 08:15:04.682410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.682382    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw" event={"ID":"c81cf60f-13a6-4260-9364-114e89edeef7","Type":"ContainerStarted","Data":"4bb56aae802847eac7f75085c548acca37e3c98bf237fc74a184b90ad1ab0580"}
Nov 02 08:15:04.685537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.685511    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv" event={"ID":"d95edc7f-4b4b-449d-870c-156648685af2","Type":"ContainerStarted","Data":"f5c264d77e8b72e454a1e404a3e460276b5bcf6d81fefa60366604b3223777e5"}
Nov 02 08:15:04.697967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.687738    4898 generic.go:356] "Generic (PLEG): container finished" podID="d8186095-8ca2-42ab-aedf-d0d44811258a" containerID="d3a006b072469b48570589117950afaf86eca3f09d182d80cf5257ecf9dbe740" exitCode=60
Nov 02 08:15:04.701544 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.699594    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc" event={"ID":"d8186095-8ca2-42ab-aedf-d0d44811258a","Type":"ContainerDied","Data":"d3a006b072469b48570589117950afaf86eca3f09d182d80cf5257ecf9dbe740"}
Nov 02 08:15:04.701544 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.700137    4898 scope.go:122] "RemoveContainer" containerID="d3a006b072469b48570589117950afaf86eca3f09d182d80cf5257ecf9dbe740"
Nov 02 08:15:04.750353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.738203    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq" event={"ID":"c3200858-4b8b-4500-bc2f-8fa185f752f6","Type":"ContainerStarted","Data":"97cad9c069e0106eec36542775f9d81c605a774ec62e154dd8131a4f6d375d39"}
Nov 02 08:15:04.801913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.789758    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l" event={"ID":"a2876d3b-6372-4dd1-919d-4a53af9540bb","Type":"ContainerStarted","Data":"7a0de784629fa01953e43776b785f65f68ff68b35a72a9469750ee79e57d5459"}
Nov 02 08:15:04.801913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.796242    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7" event={"ID":"7ebd6238-df9e-4a53-8f56-66b32a3167fd","Type":"ContainerStarted","Data":"08b479e355f416c341789e5f62fc727cabb38c32b1b6565818576ff514caf3ee"}
Nov 02 08:15:04.801913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.798928    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48" event={"ID":"e2da54f0-7885-4cd9-969c-c6051ba28f41","Type":"ContainerStarted","Data":"3bd1c594785c5ef21aa5fd6906e6faa69abf06b9fca4420f02b0508c23103180"}
Nov 02 08:15:04.801913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.801721    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88" event={"ID":"ee823ad5-4625-4083-9cc9-20044043dedf","Type":"ContainerStarted","Data":"7c40d8f1825968d4ddaf369cb71f46ce504ab6ff36e2362bed5f70f8a37fbdbb"}
Nov 02 08:15:04.803900 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.803863    4898 generic.go:356] "Generic (PLEG): container finished" podID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" containerID="f76ad9d5e3c338a4ef8d979aaf529eb4962647502e3692747cd2de47be07125e" exitCode=60
Nov 02 08:15:04.804096 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.804000    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz" event={"ID":"a6cfc1ff-3ea1-4845-98c2-d02d836507a2","Type":"ContainerDied","Data":"f76ad9d5e3c338a4ef8d979aaf529eb4962647502e3692747cd2de47be07125e"}
Nov 02 08:15:04.804148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.804137    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz" event={"ID":"a6cfc1ff-3ea1-4845-98c2-d02d836507a2","Type":"ContainerStarted","Data":"8a5f9b664d5cebc5bc2259d5986151db0a410fcd3290856016fcfdc1cced9a27"}
Nov 02 08:15:04.807169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.807146    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6" event={"ID":"9584c1c3-65cf-4e75-aa0f-9a3681010cc2","Type":"ContainerStarted","Data":"d3c9d94a1cd8a047ebb40ed734f8029118dad50ff6aff3250e7e9a29ada87b5f"}
Nov 02 08:15:04.809418 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.809398    4898 generic.go:356] "Generic (PLEG): container finished" podID="13ef0f94-811c-4542-aac5-f4ba52c18f27" containerID="3b8588fa7d62954c1426d8b142b8c8f65dd02d7b79416435de0dd4cedb478d9c" exitCode=60
Nov 02 08:15:04.809512 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.809462    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd" event={"ID":"13ef0f94-811c-4542-aac5-f4ba52c18f27","Type":"ContainerDied","Data":"3b8588fa7d62954c1426d8b142b8c8f65dd02d7b79416435de0dd4cedb478d9c"}
Nov 02 08:15:04.809749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.809731    4898 scope.go:122] "RemoveContainer" containerID="3b8588fa7d62954c1426d8b142b8c8f65dd02d7b79416435de0dd4cedb478d9c"
Nov 02 08:15:04.817931 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.813742    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff" event={"ID":"0615ef0f-a1d8-47cc-a6b4-481fadc306bc","Type":"ContainerStarted","Data":"d559ed4b90b4fcf569b2a38aa7c3ecc3d2ebf61d10ce98bad3148c41668ab73b"}
Nov 02 08:15:04.828390 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.828348    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv" podStartSLOduration=4.828335402 podStartE2EDuration="4.828335402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:04.825145514 +0000 UTC m=+3055.515472479" watchObservedRunningTime="2026-11-02 08:15:04.828335402 +0000 UTC m=+3055.518662366"
Nov 02 08:15:04.994733 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:04.994664    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88" podStartSLOduration=4.994650191 podStartE2EDuration="4.994650191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:04.993263784 +0000 UTC m=+3055.683590749" watchObservedRunningTime="2026-11-02 08:15:04.994650191 +0000 UTC m=+3055.684977154"
Nov 02 08:15:05.167378 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.167320    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff" podStartSLOduration=5.167299463 podStartE2EDuration="5.167299463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:05.164603731 +0000 UTC m=+3055.854930698" watchObservedRunningTime="2026-11-02 08:15:05.167299463 +0000 UTC m=+3055.857626427"
Nov 02 08:15:05.209459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.209403    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz" podStartSLOduration=5.209390094 podStartE2EDuration="5.209390094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:05.204243633 +0000 UTC m=+3055.894570599" watchObservedRunningTime="2026-11-02 08:15:05.209390094 +0000 UTC m=+3055.899717058"
Nov 02 08:15:05.827580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.827545    4898 generic.go:356] "Generic (PLEG): container finished" podID="c3200858-4b8b-4500-bc2f-8fa185f752f6" containerID="97cad9c069e0106eec36542775f9d81c605a774ec62e154dd8131a4f6d375d39" exitCode=60
Nov 02 08:15:05.828109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.827625    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq" event={"ID":"c3200858-4b8b-4500-bc2f-8fa185f752f6","Type":"ContainerDied","Data":"97cad9c069e0106eec36542775f9d81c605a774ec62e154dd8131a4f6d375d39"}
Nov 02 08:15:05.828109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.827697    4898 scope.go:122] "RemoveContainer" containerID="bcec8ffcf60e191504f8bca6c6a7d6b685ce9e11c4763d424034d9f2a54a8840"
Nov 02 08:15:05.828109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.827978    4898 scope.go:122] "RemoveContainer" containerID="97cad9c069e0106eec36542775f9d81c605a774ec62e154dd8131a4f6d375d39"
Nov 02 08:15:05.828445 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.828421    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq_odh-ai-gateway-infra(c3200858-4b8b-4500-bc2f-8fa185f752f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6"
Nov 02 08:15:05.830119 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.830095    4898 generic.go:356] "Generic (PLEG): container finished" podID="a2876d3b-6372-4dd1-919d-4a53af9540bb" containerID="7a0de784629fa01953e43776b785f65f68ff68b35a72a9469750ee79e57d5459" exitCode=60
Nov 02 08:15:05.830213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.830172    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l" event={"ID":"a2876d3b-6372-4dd1-919d-4a53af9540bb","Type":"ContainerDied","Data":"7a0de784629fa01953e43776b785f65f68ff68b35a72a9469750ee79e57d5459"}
Nov 02 08:15:05.830443 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.830427    4898 scope.go:122] "RemoveContainer" containerID="7a0de784629fa01953e43776b785f65f68ff68b35a72a9469750ee79e57d5459"
Nov 02 08:15:05.830797 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.830775    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l_odh-ai-gateway-infra(a2876d3b-6372-4dd1-919d-4a53af9540bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb"
Nov 02 08:15:05.832329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.832303    4898 generic.go:356] "Generic (PLEG): container finished" podID="7bca6afb-be58-48e6-8d4f-317c45edcffd" containerID="8109cab730c32f848447f5f3d21d601cc8615c8d619fffc4fb801b2a283c0bb8" exitCode=60
Nov 02 08:15:05.832735 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.832393    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8" event={"ID":"7bca6afb-be58-48e6-8d4f-317c45edcffd","Type":"ContainerDied","Data":"8109cab730c32f848447f5f3d21d601cc8615c8d619fffc4fb801b2a283c0bb8"}
Nov 02 08:15:05.832735 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.832719    4898 scope.go:122] "RemoveContainer" containerID="8109cab730c32f848447f5f3d21d601cc8615c8d619fffc4fb801b2a283c0bb8"
Nov 02 08:15:05.833127 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.833102    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8_odh-ai-gateway-infra(7bca6afb-be58-48e6-8d4f-317c45edcffd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd"
Nov 02 08:15:05.834476 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.834455    4898 generic.go:356] "Generic (PLEG): container finished" podID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" containerID="08b479e355f416c341789e5f62fc727cabb38c32b1b6565818576ff514caf3ee" exitCode=60
Nov 02 08:15:05.835129 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.834535    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7" event={"ID":"7ebd6238-df9e-4a53-8f56-66b32a3167fd","Type":"ContainerDied","Data":"08b479e355f416c341789e5f62fc727cabb38c32b1b6565818576ff514caf3ee"}
Nov 02 08:15:05.835129 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.834763    4898 scope.go:122] "RemoveContainer" containerID="08b479e355f416c341789e5f62fc727cabb38c32b1b6565818576ff514caf3ee"
Nov 02 08:15:05.835129 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.835103    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7_odh-ai-gateway-infra(7ebd6238-df9e-4a53-8f56-66b32a3167fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd"
Nov 02 08:15:05.836488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.836467    4898 generic.go:356] "Generic (PLEG): container finished" podID="e2da54f0-7885-4cd9-969c-c6051ba28f41" containerID="3bd1c594785c5ef21aa5fd6906e6faa69abf06b9fca4420f02b0508c23103180" exitCode=60
Nov 02 08:15:05.836575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.836545    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48" event={"ID":"e2da54f0-7885-4cd9-969c-c6051ba28f41","Type":"ContainerDied","Data":"3bd1c594785c5ef21aa5fd6906e6faa69abf06b9fca4420f02b0508c23103180"}
Nov 02 08:15:05.836797 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.836780    4898 scope.go:122] "RemoveContainer" containerID="3bd1c594785c5ef21aa5fd6906e6faa69abf06b9fca4420f02b0508c23103180"
Nov 02 08:15:05.837175 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.837145    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48_odh-ai-gateway-infra(e2da54f0-7885-4cd9-969c-c6051ba28f41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41"
Nov 02 08:15:05.838534 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.838515    4898 generic.go:356] "Generic (PLEG): container finished" podID="ee823ad5-4625-4083-9cc9-20044043dedf" containerID="7c40d8f1825968d4ddaf369cb71f46ce504ab6ff36e2362bed5f70f8a37fbdbb" exitCode=60
Nov 02 08:15:05.838617 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.838594    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88" event={"ID":"ee823ad5-4625-4083-9cc9-20044043dedf","Type":"ContainerDied","Data":"7c40d8f1825968d4ddaf369cb71f46ce504ab6ff36e2362bed5f70f8a37fbdbb"}
Nov 02 08:15:05.838831 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.838816    4898 scope.go:122] "RemoveContainer" containerID="7c40d8f1825968d4ddaf369cb71f46ce504ab6ff36e2362bed5f70f8a37fbdbb"
Nov 02 08:15:05.839132 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.839115    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88_odh-ai-gateway-infra(ee823ad5-4625-4083-9cc9-20044043dedf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88" podUID="ee823ad5-4625-4083-9cc9-20044043dedf"
Nov 02 08:15:05.840452 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.840434    4898 generic.go:356] "Generic (PLEG): container finished" podID="fab5290b-4dd7-494c-9381-a467e78f96c8" containerID="2af11f40f2476b3614447ea5f580189cc5e26d3de23ec6a12b342fcde618990a" exitCode=60
Nov 02 08:15:05.840568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.840498    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq" event={"ID":"fab5290b-4dd7-494c-9381-a467e78f96c8","Type":"ContainerDied","Data":"2af11f40f2476b3614447ea5f580189cc5e26d3de23ec6a12b342fcde618990a"}
Nov 02 08:15:05.840736 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.840723    4898 scope.go:122] "RemoveContainer" containerID="2af11f40f2476b3614447ea5f580189cc5e26d3de23ec6a12b342fcde618990a"
Nov 02 08:15:05.841065 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.841045    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq_odh-ai-gateway-infra(fab5290b-4dd7-494c-9381-a467e78f96c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8"
Nov 02 08:15:05.842458 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.842438    4898 generic.go:356] "Generic (PLEG): container finished" podID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" containerID="8a5f9b664d5cebc5bc2259d5986151db0a410fcd3290856016fcfdc1cced9a27" exitCode=60
Nov 02 08:15:05.842556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.842516    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz" event={"ID":"a6cfc1ff-3ea1-4845-98c2-d02d836507a2","Type":"ContainerDied","Data":"8a5f9b664d5cebc5bc2259d5986151db0a410fcd3290856016fcfdc1cced9a27"}
Nov 02 08:15:05.842732 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.842711    4898 scope.go:122] "RemoveContainer" containerID="8a5f9b664d5cebc5bc2259d5986151db0a410fcd3290856016fcfdc1cced9a27"
Nov 02 08:15:05.843119 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.843085    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz_odh-ai-gateway-infra(a6cfc1ff-3ea1-4845-98c2-d02d836507a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2"
Nov 02 08:15:05.844933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.844841    4898 generic.go:356] "Generic (PLEG): container finished" podID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" containerID="d3c9d94a1cd8a047ebb40ed734f8029118dad50ff6aff3250e7e9a29ada87b5f" exitCode=60
Nov 02 08:15:05.845044 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.845002    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6" event={"ID":"9584c1c3-65cf-4e75-aa0f-9a3681010cc2","Type":"ContainerDied","Data":"d3c9d94a1cd8a047ebb40ed734f8029118dad50ff6aff3250e7e9a29ada87b5f"}
Nov 02 08:15:05.845162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.845141    4898 scope.go:122] "RemoveContainer" containerID="d3c9d94a1cd8a047ebb40ed734f8029118dad50ff6aff3250e7e9a29ada87b5f"
Nov 02 08:15:05.845554 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.845530    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6_odh-ai-gateway-infra(9584c1c3-65cf-4e75-aa0f-9a3681010cc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2"
Nov 02 08:15:05.847328 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.847306    4898 generic.go:356] "Generic (PLEG): container finished" podID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" containerID="d559ed4b90b4fcf569b2a38aa7c3ecc3d2ebf61d10ce98bad3148c41668ab73b" exitCode=60
Nov 02 08:15:05.847419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.847350    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff" event={"ID":"0615ef0f-a1d8-47cc-a6b4-481fadc306bc","Type":"ContainerDied","Data":"d559ed4b90b4fcf569b2a38aa7c3ecc3d2ebf61d10ce98bad3148c41668ab73b"}
Nov 02 08:15:05.847633 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.847614    4898 scope.go:122] "RemoveContainer" containerID="d559ed4b90b4fcf569b2a38aa7c3ecc3d2ebf61d10ce98bad3148c41668ab73b"
Nov 02 08:15:05.847953 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.847932    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff_odh-ai-gateway-infra(0615ef0f-a1d8-47cc-a6b4-481fadc306bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc"
Nov 02 08:15:05.850111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.850089    4898 generic.go:356] "Generic (PLEG): container finished" podID="1f648095-5449-42df-885a-29e8044cb8c2" containerID="8fd44eb5808145b94b4867fe641a9f8688b4a9e0ae70903069b5befc3ea90734" exitCode=60
Nov 02 08:15:05.850226 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.850167    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv" event={"ID":"1f648095-5449-42df-885a-29e8044cb8c2","Type":"ContainerDied","Data":"8fd44eb5808145b94b4867fe641a9f8688b4a9e0ae70903069b5befc3ea90734"}
Nov 02 08:15:05.850408 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.850391    4898 scope.go:122] "RemoveContainer" containerID="8fd44eb5808145b94b4867fe641a9f8688b4a9e0ae70903069b5befc3ea90734"
Nov 02 08:15:05.850672 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.850647    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv_odh-ai-gateway-infra(1f648095-5449-42df-885a-29e8044cb8c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv" podUID="1f648095-5449-42df-885a-29e8044cb8c2"
Nov 02 08:15:05.852301 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.852277    4898 generic.go:356] "Generic (PLEG): container finished" podID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" containerID="18f9173ea272240ed1bdcfb60555a5e221f08c491d89d449a8290b2fcd44ae06" exitCode=60
Nov 02 08:15:05.852401 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.852362    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz" event={"ID":"5ff46f33-1e5a-4a41-85a1-7d18107f3844","Type":"ContainerDied","Data":"18f9173ea272240ed1bdcfb60555a5e221f08c491d89d449a8290b2fcd44ae06"}
Nov 02 08:15:05.852597 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.852582    4898 scope.go:122] "RemoveContainer" containerID="18f9173ea272240ed1bdcfb60555a5e221f08c491d89d449a8290b2fcd44ae06"
Nov 02 08:15:05.852820 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.852799    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz_odh-ai-gateway-infra(5ff46f33-1e5a-4a41-85a1-7d18107f3844)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844"
Nov 02 08:15:05.857139 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.857115    4898 scope.go:122] "RemoveContainer" containerID="fb6e368de6fdd8eccab236788c0e2e246ae090a07809b7c561d167c5ce29c6a8"
Nov 02 08:15:05.858396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.858370    4898 generic.go:356] "Generic (PLEG): container finished" podID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" containerID="b9835992b195fe924a36266980d50468e3af07516fa0ddaf46cf388966b6e7ed" exitCode=60
Nov 02 08:15:05.858521 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.858454    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s" event={"ID":"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5","Type":"ContainerDied","Data":"b9835992b195fe924a36266980d50468e3af07516fa0ddaf46cf388966b6e7ed"}
Nov 02 08:15:05.858822 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.858802    4898 scope.go:122] "RemoveContainer" containerID="b9835992b195fe924a36266980d50468e3af07516fa0ddaf46cf388966b6e7ed"
Nov 02 08:15:05.859211 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.859125    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s_odh-ai-gateway-infra(abf3abb8-f4bd-4770-8ce7-5df5defd7fb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5"
Nov 02 08:15:05.862789 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.862765    4898 generic.go:356] "Generic (PLEG): container finished" podID="c81cf60f-13a6-4260-9364-114e89edeef7" containerID="4bb56aae802847eac7f75085c548acca37e3c98bf237fc74a184b90ad1ab0580" exitCode=60
Nov 02 08:15:05.862908 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.862847    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw" event={"ID":"c81cf60f-13a6-4260-9364-114e89edeef7","Type":"ContainerDied","Data":"4bb56aae802847eac7f75085c548acca37e3c98bf237fc74a184b90ad1ab0580"}
Nov 02 08:15:05.863101 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.863084    4898 scope.go:122] "RemoveContainer" containerID="4bb56aae802847eac7f75085c548acca37e3c98bf237fc74a184b90ad1ab0580"
Nov 02 08:15:05.863424 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.863397    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw_odh-ai-gateway-infra(c81cf60f-13a6-4260-9364-114e89edeef7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw" podUID="c81cf60f-13a6-4260-9364-114e89edeef7"
Nov 02 08:15:05.865185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.865162    4898 generic.go:356] "Generic (PLEG): container finished" podID="d95edc7f-4b4b-449d-870c-156648685af2" containerID="f5c264d77e8b72e454a1e404a3e460276b5bcf6d81fefa60366604b3223777e5" exitCode=60
Nov 02 08:15:05.865303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.865202    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv" event={"ID":"d95edc7f-4b4b-449d-870c-156648685af2","Type":"ContainerDied","Data":"f5c264d77e8b72e454a1e404a3e460276b5bcf6d81fefa60366604b3223777e5"}
Nov 02 08:15:05.865476 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.865459    4898 scope.go:122] "RemoveContainer" containerID="f5c264d77e8b72e454a1e404a3e460276b5bcf6d81fefa60366604b3223777e5"
Nov 02 08:15:05.865732 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:05.865711    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv_odh-ai-gateway-infra(d95edc7f-4b4b-449d-870c-156648685af2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv" podUID="d95edc7f-4b4b-449d-870c-156648685af2"
Nov 02 08:15:05.892972 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.892941    4898 scope.go:122] "RemoveContainer" containerID="af3e69eee79f3cf15ea9b39a65b0e4ffee06488ef3a103bbee878b0b0aecc770"
Nov 02 08:15:05.910235 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.910210    4898 scope.go:122] "RemoveContainer" containerID="c79171882ad8e7c54b9506a696083134c863d0dce72238acd335a35c7b056bb0"
Nov 02 08:15:05.926027 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.926007    4898 scope.go:122] "RemoveContainer" containerID="b97ae07ee35983f66ac1f2df056400b4203a5039039c66f1fe8c7312d86ef9ee"
Nov 02 08:15:05.940803 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.940779    4898 scope.go:122] "RemoveContainer" containerID="240d0cb823cc583f76316c68ee3fb756eeae7e74d6c4942a0cd4c4c2395e371c"
Nov 02 08:15:05.957427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.957405    4898 scope.go:122] "RemoveContainer" containerID="48706592fe209b45294efb6b39084cf20ca330bdbe40c8b1c5f8edc5d54ea77e"
Nov 02 08:15:05.978610 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.978579    4898 scope.go:122] "RemoveContainer" containerID="f76ad9d5e3c338a4ef8d979aaf529eb4962647502e3692747cd2de47be07125e"
Nov 02 08:15:05.998727 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:05.998698    4898 scope.go:122] "RemoveContainer" containerID="7ed7b8800bca385b0540dddee8fe8298f8d45989fd503442da0356f343d637df"
Nov 02 08:15:06.014989 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.014962    4898 scope.go:122] "RemoveContainer" containerID="84418be6ff68cedf99bdf017e9cdcb9421c72886e5b99eefc37959baae5a8876"
Nov 02 08:15:06.029677 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.029656    4898 scope.go:122] "RemoveContainer" containerID="1fd602f0dc7dbc5493ba5fccabf2d0fb7c9e6e475a75d768362ac5d9971de281"
Nov 02 08:15:06.045332 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.045309    4898 scope.go:122] "RemoveContainer" containerID="25d66d094bf9b5cdb9b247166247c23d3a0b7a95b7eb669c884937dedf6ce668"
Nov 02 08:15:06.060976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.060917    4898 scope.go:122] "RemoveContainer" containerID="f26e2d10012cb1202bf7b688a73557207f8993c7065a5ce60be30095f46709b6"
Nov 02 08:15:06.079202 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.079175    4898 scope.go:122] "RemoveContainer" containerID="4e901b40c3f01dd7eaf30f909be7fab3eadb84dda7b49dc794103ebb2b7b7852"
Nov 02 08:15:06.096991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.096963    4898 scope.go:122] "RemoveContainer" containerID="0f8a68f225d7cd7b6de960a9d7bd374ee92f7815f7eabfc1af654f4f1e399ab4"
Nov 02 08:15:06.880768 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.880733    4898 generic.go:356] "Generic (PLEG): container finished" podID="d8186095-8ca2-42ab-aedf-d0d44811258a" containerID="f7b521eb30f7e43e99cd2100de19eff81864f8be04bd8965ac30afda4d41ff3f" exitCode=60
Nov 02 08:15:06.881285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.880766    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc" event={"ID":"d8186095-8ca2-42ab-aedf-d0d44811258a","Type":"ContainerDied","Data":"f7b521eb30f7e43e99cd2100de19eff81864f8be04bd8965ac30afda4d41ff3f"}
Nov 02 08:15:06.881285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.880827    4898 scope.go:122] "RemoveContainer" containerID="d3a006b072469b48570589117950afaf86eca3f09d182d80cf5257ecf9dbe740"
Nov 02 08:15:06.881285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.881263    4898 scope.go:122] "RemoveContainer" containerID="f7b521eb30f7e43e99cd2100de19eff81864f8be04bd8965ac30afda4d41ff3f"
Nov 02 08:15:06.881746 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.881717    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc_odh-ai-gateway-infra(d8186095-8ca2-42ab-aedf-d0d44811258a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a"
Nov 02 08:15:06.884124 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.884102    4898 scope.go:122] "RemoveContainer" containerID="97cad9c069e0106eec36542775f9d81c605a774ec62e154dd8131a4f6d375d39"
Nov 02 08:15:06.884558 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.884517    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq_odh-ai-gateway-infra(c3200858-4b8b-4500-bc2f-8fa185f752f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6"
Nov 02 08:15:06.886099 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.886081    4898 scope.go:122] "RemoveContainer" containerID="7a0de784629fa01953e43776b785f65f68ff68b35a72a9469750ee79e57d5459"
Nov 02 08:15:06.886433 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.886409    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l_odh-ai-gateway-infra(a2876d3b-6372-4dd1-919d-4a53af9540bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb"
Nov 02 08:15:06.888299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.888279    4898 scope.go:122] "RemoveContainer" containerID="8109cab730c32f848447f5f3d21d601cc8615c8d619fffc4fb801b2a283c0bb8"
Nov 02 08:15:06.888752 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.888725    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8_odh-ai-gateway-infra(7bca6afb-be58-48e6-8d4f-317c45edcffd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd"
Nov 02 08:15:06.890411 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.890393    4898 scope.go:122] "RemoveContainer" containerID="08b479e355f416c341789e5f62fc727cabb38c32b1b6565818576ff514caf3ee"
Nov 02 08:15:06.890697 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.890674    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7_odh-ai-gateway-infra(7ebd6238-df9e-4a53-8f56-66b32a3167fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd"
Nov 02 08:15:06.892303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.892273    4898 scope.go:122] "RemoveContainer" containerID="3bd1c594785c5ef21aa5fd6906e6faa69abf06b9fca4420f02b0508c23103180"
Nov 02 08:15:06.892629 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.892595    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48_odh-ai-gateway-infra(e2da54f0-7885-4cd9-969c-c6051ba28f41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41"
Nov 02 08:15:06.894104 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.894082    4898 scope.go:122] "RemoveContainer" containerID="7c40d8f1825968d4ddaf369cb71f46ce504ab6ff36e2362bed5f70f8a37fbdbb"
Nov 02 08:15:06.894484 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.894454    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88_odh-ai-gateway-infra(ee823ad5-4625-4083-9cc9-20044043dedf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88" podUID="ee823ad5-4625-4083-9cc9-20044043dedf"
Nov 02 08:15:06.895968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.895948    4898 scope.go:122] "RemoveContainer" containerID="2af11f40f2476b3614447ea5f580189cc5e26d3de23ec6a12b342fcde618990a"
Nov 02 08:15:06.896235 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.896213    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq_odh-ai-gateway-infra(fab5290b-4dd7-494c-9381-a467e78f96c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8"
Nov 02 08:15:06.897881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.897860    4898 scope.go:122] "RemoveContainer" containerID="8a5f9b664d5cebc5bc2259d5986151db0a410fcd3290856016fcfdc1cced9a27"
Nov 02 08:15:06.899107 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.899076    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz_odh-ai-gateway-infra(a6cfc1ff-3ea1-4845-98c2-d02d836507a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2"
Nov 02 08:15:06.900744 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.900722    4898 scope.go:122] "RemoveContainer" containerID="d3c9d94a1cd8a047ebb40ed734f8029118dad50ff6aff3250e7e9a29ada87b5f"
Nov 02 08:15:06.901140 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.901108    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6_odh-ai-gateway-infra(9584c1c3-65cf-4e75-aa0f-9a3681010cc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2"
Nov 02 08:15:06.903945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.903808    4898 generic.go:356] "Generic (PLEG): container finished" podID="13ef0f94-811c-4542-aac5-f4ba52c18f27" containerID="488f972f6b1eb8e70830222a250dc62e6d9cafc7ae44ccf3d3f2f1f97c106f63" exitCode=60
Nov 02 08:15:06.904058 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.903932    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd" event={"ID":"13ef0f94-811c-4542-aac5-f4ba52c18f27","Type":"ContainerDied","Data":"488f972f6b1eb8e70830222a250dc62e6d9cafc7ae44ccf3d3f2f1f97c106f63"}
Nov 02 08:15:06.904312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.904295    4898 scope.go:122] "RemoveContainer" containerID="488f972f6b1eb8e70830222a250dc62e6d9cafc7ae44ccf3d3f2f1f97c106f63"
Nov 02 08:15:06.904625 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.904596    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd_odh-ai-gateway-infra(13ef0f94-811c-4542-aac5-f4ba52c18f27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27"
Nov 02 08:15:06.906314 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.906289    4898 scope.go:122] "RemoveContainer" containerID="d559ed4b90b4fcf569b2a38aa7c3ecc3d2ebf61d10ce98bad3148c41668ab73b"
Nov 02 08:15:06.906604 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.906578    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff_odh-ai-gateway-infra(0615ef0f-a1d8-47cc-a6b4-481fadc306bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc"
Nov 02 08:15:06.908377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.908356    4898 scope.go:122] "RemoveContainer" containerID="8fd44eb5808145b94b4867fe641a9f8688b4a9e0ae70903069b5befc3ea90734"
Nov 02 08:15:06.908678 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.908645    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv_odh-ai-gateway-infra(1f648095-5449-42df-885a-29e8044cb8c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv" podUID="1f648095-5449-42df-885a-29e8044cb8c2"
Nov 02 08:15:06.910097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.910074    4898 scope.go:122] "RemoveContainer" containerID="3b8588fa7d62954c1426d8b142b8c8f65dd02d7b79416435de0dd4cedb478d9c"
Nov 02 08:15:06.911013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.910879    4898 scope.go:122] "RemoveContainer" containerID="18f9173ea272240ed1bdcfb60555a5e221f08c491d89d449a8290b2fcd44ae06"
Nov 02 08:15:06.911273 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.911250    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz_odh-ai-gateway-infra(5ff46f33-1e5a-4a41-85a1-7d18107f3844)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844"
Nov 02 08:15:06.912765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.912747    4898 scope.go:122] "RemoveContainer" containerID="b9835992b195fe924a36266980d50468e3af07516fa0ddaf46cf388966b6e7ed"
Nov 02 08:15:06.913061 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.913042    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s_odh-ai-gateway-infra(abf3abb8-f4bd-4770-8ce7-5df5defd7fb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5"
Nov 02 08:15:06.914632 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.914613    4898 scope.go:122] "RemoveContainer" containerID="4bb56aae802847eac7f75085c548acca37e3c98bf237fc74a184b90ad1ab0580"
Nov 02 08:15:06.914866 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.914846    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw_odh-ai-gateway-infra(c81cf60f-13a6-4260-9364-114e89edeef7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw" podUID="c81cf60f-13a6-4260-9364-114e89edeef7"
Nov 02 08:15:06.916299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:06.916282    4898 scope.go:122] "RemoveContainer" containerID="f5c264d77e8b72e454a1e404a3e460276b5bcf6d81fefa60366604b3223777e5"
Nov 02 08:15:06.916511 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:06.916493    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv_odh-ai-gateway-infra(d95edc7f-4b4b-449d-870c-156648685af2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv" podUID="d95edc7f-4b4b-449d-870c-156648685af2"
Nov 02 08:15:07.928024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:07.927992    4898 scope.go:122] "RemoveContainer" containerID="f7b521eb30f7e43e99cd2100de19eff81864f8be04bd8965ac30afda4d41ff3f"
Nov 02 08:15:07.928501 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:07.928382    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc_odh-ai-gateway-infra(d8186095-8ca2-42ab-aedf-d0d44811258a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a"
Nov 02 08:15:07.929784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:07.929766    4898 scope.go:122] "RemoveContainer" containerID="488f972f6b1eb8e70830222a250dc62e6d9cafc7ae44ccf3d3f2f1f97c106f63"
Nov 02 08:15:07.930076 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:07.930054    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd_odh-ai-gateway-infra(13ef0f94-811c-4542-aac5-f4ba52c18f27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27"
Nov 02 08:15:09.841093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.841052    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"]
Nov 02 08:15:09.849915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.848086    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.919826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.891138    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-certs\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.919826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.891338    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-data\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.919826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.891665    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-envoy\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.919826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.891715    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcm68\" (UniqueName: \"kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-kube-api-access-pcm68\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.919826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.892032    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb263662-7712-4909-92c9-deb40f800844-istio-podinfo\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.919826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.892155    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-socket\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.919826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.892260    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-credential-socket\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.919826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.892323    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb263662-7712-4909-92c9-deb40f800844-istiod-ca-cert\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.919826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.892437    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-istio-token\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.919826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.894182    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f765b8-openshift-default-dockercfg-gws7z\""
Nov 02 08:15:09.920715 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.920280    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"]
Nov 02 08:15:09.995382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.995117    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb263662-7712-4909-92c9-deb40f800844-istio-podinfo\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.997219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.997185    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-socket\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.997438 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.997251    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-credential-socket\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.997438 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.997291    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb263662-7712-4909-92c9-deb40f800844-istiod-ca-cert\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.997438 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.997397    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-istio-token\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.997696 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.997468    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-certs\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.997696 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.997627    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-data\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.997696 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.997648    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-socket\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.997696 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.997657    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-envoy\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.998093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.997755    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcm68\" (UniqueName: \"kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-kube-api-access-pcm68\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.998503 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.998473    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-credential-socket\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.998636 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.998566    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-certs\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.998757 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.998732    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-data\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:09.999319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:09.999294    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb263662-7712-4909-92c9-deb40f800844-istiod-ca-cert\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:10.049930 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:10.047627    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb263662-7712-4909-92c9-deb40f800844-istio-podinfo\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:10.066701 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:10.058536    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-istio-token\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:10.074364 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:10.074326    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-envoy\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:10.115957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:10.092635    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcm68\" (UniqueName: \"kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-kube-api-access-pcm68\") pod \"e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") " pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:10.197429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:10.197388    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:10.651514 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:10.651481    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb263662_7712_4909_92c9_deb40f800844.slice/crio-535b2819c0f4465b213febefcbbb0684c2215776d9ddd9462b27be2b74060e82 WatchSource:0}: Error finding container 535b2819c0f4465b213febefcbbb0684c2215776d9ddd9462b27be2b74060e82: Status 404 returned error can't find the container with id 535b2819c0f4465b213febefcbbb0684c2215776d9ddd9462b27be2b74060e82
Nov 02 08:15:10.654515 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:10.654483    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"]
Nov 02 08:15:10.657259 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:10.657209    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:15:10.657377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:10.657312    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:15:10.657377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:10.657363    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:15:11.006235 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:11.006191    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" event={"ID":"cb263662-7712-4909-92c9-deb40f800844","Type":"ContainerStarted","Data":"d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6"}
Nov 02 08:15:11.006643 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:11.006265    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" event={"ID":"cb263662-7712-4909-92c9-deb40f800844","Type":"ContainerStarted","Data":"535b2819c0f4465b213febefcbbb0684c2215776d9ddd9462b27be2b74060e82"}
Nov 02 08:15:11.073353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:11.073290    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" podStartSLOduration=2.0732685 podStartE2EDuration="2.0732685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:11.065799312 +0000 UTC m=+3061.756126303" watchObservedRunningTime="2026-11-02 08:15:11.0732685 +0000 UTC m=+3061.763595465"
Nov 02 08:15:11.205967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:11.205280    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:11.220921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:11.219039    4898 patch_prober.go:29] interesting pod/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Nov 02 08:15:11.220921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:11.219111    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Nov 02 08:15:12.039461 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:12.039427    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"]
Nov 02 08:15:12.095918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:12.095593    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec18d4e9-h57bb"]
Nov 02 08:15:12.204226 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:12.204195    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:13.025643 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:13.025608    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:13.026778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:13.026755    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:15:13.552196 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:13.552144    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c328dd-8075-4654-8b66-24920ae84817" path="/var/lib/kubelet/pods/58c328dd-8075-4654-8b66-24920ae84817/volumes"
Nov 02 08:15:13.858979 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:13.856878    4898 scope.go:122] "RemoveContainer" containerID="9d81230def02a4f54da680b0cc7ac91ef6aea86239dc039052c24e9e28496035"
Nov 02 08:15:13.896261 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:13.896219    4898 scope.go:122] "RemoveContainer" containerID="0a68323103298ef27517f4d52563174b3bcfac83eb9c73c81c40b71daeb0bf03"
Nov 02 08:15:13.916729 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:13.916705    4898 scope.go:122] "RemoveContainer" containerID="f7d48ae6e93dd893a069deb2816b4fa32a388aff0345cb11b59c5433ba57966e"
Nov 02 08:15:13.930863 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:13.930830    4898 scope.go:122] "RemoveContainer" containerID="1d1247b2cbc987e78cc9dca0f2fbfcb7a5121e85a6b83bb7bd3cf649c3865d9b"
Nov 02 08:15:15.434184 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:15.427092    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:15:15.441675 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:15.441629    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"]
Nov 02 08:15:15.450911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:15.449037    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"
Nov 02 08:15:15.501940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:15.480664    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f765b8-dockercfg-2nstz\""
Nov 02 08:15:15.501940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:15.496127    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f765b8\""
Nov 02 08:15:15.648291 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:15.647403    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvqjb\" (UniqueName: \"kubernetes.io/projected/30f79c15-0f1d-4041-858c-07252b72704e-kube-api-access-fvqjb\") pod \"payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j\" (UID: \"30f79c15-0f1d-4041-858c-07252b72704e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"
Nov 02 08:15:15.648291 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:15.647483    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f79c15-0f1d-4041-858c-07252b72704e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j\" (UID: \"30f79c15-0f1d-4041-858c-07252b72704e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"
Nov 02 08:15:15.779973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:15.755165    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"]
Nov 02 08:15:15.896498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:15.878194    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvqjb\" (UniqueName: \"kubernetes.io/projected/30f79c15-0f1d-4041-858c-07252b72704e-kube-api-access-fvqjb\") pod \"payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j\" (UID: \"30f79c15-0f1d-4041-858c-07252b72704e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"
Nov 02 08:15:15.896498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:15.878274    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f79c15-0f1d-4041-858c-07252b72704e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j\" (UID: \"30f79c15-0f1d-4041-858c-07252b72704e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"
Nov 02 08:15:15.896498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:15.879433    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f79c15-0f1d-4041-858c-07252b72704e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j\" (UID: \"30f79c15-0f1d-4041-858c-07252b72704e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"
Nov 02 08:15:15.935023 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:15.934982    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvqjb\" (UniqueName: \"kubernetes.io/projected/30f79c15-0f1d-4041-858c-07252b72704e-kube-api-access-fvqjb\") pod \"payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j\" (UID: \"30f79c15-0f1d-4041-858c-07252b72704e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"
Nov 02 08:15:15.987404 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:15.987181    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"
Nov 02 08:15:16.132362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.132328    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"]
Nov 02 08:15:16.137879 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.137853    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.179915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.178371    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"agc-cred-f4251183894c\""
Nov 02 08:15:16.179915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.178553    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"kube-root-ca.crt\""
Nov 02 08:15:16.179915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.178657    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"routing-overlay\""
Nov 02 08:15:16.179915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.178715    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"agc-cred-52831c4c288c\""
Nov 02 08:15:16.179915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.179374    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"openshift-service-ca.crt\""
Nov 02 08:15:16.179915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.179442    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ea9556\""
Nov 02 08:15:16.196539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.196093    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ea9556\"/\"payload-processing-external-model-e2e--c1e7f36e-dockercfg-52r9d\""
Nov 02 08:15:16.197272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.197235    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.197418 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.197347    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6drx4\" (UniqueName: \"kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-kube-api-access-6drx4\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.197482 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.197434    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.197482 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.197463    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.203645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.203616    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"]
Nov 02 08:15:16.319823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.313179    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.319823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.313318    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6drx4\" (UniqueName: \"kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-kube-api-access-6drx4\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.319823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.313406    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.319823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.313443    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.319823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.316147    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.333174 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.322461    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.335606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.335575    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.339069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.339044    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6drx4\" (UniqueName: \"kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-kube-api-access-6drx4\") pod \"payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") " pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.396751 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.396722    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"]
Nov 02 08:15:16.475421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.474196    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:16.478909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.476868    4898 scope.go:122] "RemoveContainer" containerID="18f9173ea272240ed1bdcfb60555a5e221f08c491d89d449a8290b2fcd44ae06"
Nov 02 08:15:16.478909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.477423    4898 scope.go:122] "RemoveContainer" containerID="4bb56aae802847eac7f75085c548acca37e3c98bf237fc74a184b90ad1ab0580"
Nov 02 08:15:16.478909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.477974    4898 scope.go:122] "RemoveContainer" containerID="d3c9d94a1cd8a047ebb40ed734f8029118dad50ff6aff3250e7e9a29ada87b5f"
Nov 02 08:15:16.478909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.478372    4898 scope.go:122] "RemoveContainer" containerID="3bd1c594785c5ef21aa5fd6906e6faa69abf06b9fca4420f02b0508c23103180"
Nov 02 08:15:16.478909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.478801    4898 scope.go:122] "RemoveContainer" containerID="2af11f40f2476b3614447ea5f580189cc5e26d3de23ec6a12b342fcde618990a"
Nov 02 08:15:16.479286 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.479161    4898 scope.go:122] "RemoveContainer" containerID="7c40d8f1825968d4ddaf369cb71f46ce504ab6ff36e2362bed5f70f8a37fbdbb"
Nov 02 08:15:16.492250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.479842    4898 scope.go:122] "RemoveContainer" containerID="b9835992b195fe924a36266980d50468e3af07516fa0ddaf46cf388966b6e7ed"
Nov 02 08:15:16.492250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:16.480563    4898 scope.go:122] "RemoveContainer" containerID="7a0de784629fa01953e43776b785f65f68ff68b35a72a9469750ee79e57d5459"
Nov 02 08:15:17.009541 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.009505    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"]
Nov 02 08:15:17.014872 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.014851    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.029675 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.029582    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.029820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.029740    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.030330 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.029939    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.030330 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.030093    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gw7t\" (UniqueName: \"kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-kube-api-access-9gw7t\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.033183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.033157    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"]
Nov 02 08:15:17.035216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.035199    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-71a41c\"/\"agc-cred-1537c015d4c3\""
Nov 02 08:15:17.039124 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.039100    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"]
Nov 02 08:15:17.145660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.145590    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gw7t\" (UniqueName: \"kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-kube-api-access-9gw7t\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.229249 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.229215    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.229615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.229589    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.229814 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.229797    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.230330 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.230259    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.230504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.230478    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.232641 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.232613    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j" event={"ID":"30f79c15-0f1d-4041-858c-07252b72704e","Type":"ContainerStarted","Data":"e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597"}
Nov 02 08:15:17.232746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.232671    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j" event={"ID":"30f79c15-0f1d-4041-858c-07252b72704e","Type":"ContainerStarted","Data":"2543b9787772195fa284d66c9f1d6de8529773bfa535515f88ecdc90972262e9"}
Nov 02 08:15:17.232746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.232721    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"
Nov 02 08:15:17.277950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.276723    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gw7t\" (UniqueName: \"kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-kube-api-access-9gw7t\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.301945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.301853    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") " pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.304685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.304329    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j" podStartSLOduration=2.304314406 podStartE2EDuration="2.304314406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:17.277996949 +0000 UTC m=+3067.968323931" watchObservedRunningTime="2026-11-02 08:15:17.304314406 +0000 UTC m=+3067.994641365"
Nov 02 08:15:17.344904 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.344858    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:17.493986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.472695    4898 scope.go:122] "RemoveContainer" containerID="8fd44eb5808145b94b4867fe641a9f8688b4a9e0ae70903069b5befc3ea90734"
Nov 02 08:15:17.493986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.482500    4898 scope.go:122] "RemoveContainer" containerID="08b479e355f416c341789e5f62fc727cabb38c32b1b6565818576ff514caf3ee"
Nov 02 08:15:17.874045 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:17.874008    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"]
Nov 02 08:15:18.009013 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:18.008971    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff85201_d3aa_4dcc_ae86_2a7865b44d82.slice/crio-bc3e8bb4321913b603978592cda3116eb6a6867037f2c750f5e44d1961a27f19 WatchSource:0}: Error finding container bc3e8bb4321913b603978592cda3116eb6a6867037f2c750f5e44d1961a27f19: Status 404 returned error can't find the container with id bc3e8bb4321913b603978592cda3116eb6a6867037f2c750f5e44d1961a27f19
Nov 02 08:15:18.249137 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.249072    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr" event={"ID":"074fba4b-7b02-46cf-9aab-4a57a38968d1","Type":"ContainerStarted","Data":"8da540b60c0d4ab101545d760ee0e4a344976550718d171672b2248f6e2e498e"}
Nov 02 08:15:18.249308 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.249142    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr" event={"ID":"074fba4b-7b02-46cf-9aab-4a57a38968d1","Type":"ContainerStarted","Data":"6ee93631ea3269953730ff1e98d04bcc1fc1c9146c57ecc0b27c015a329fb743"}
Nov 02 08:15:18.249489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.249465    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:18.251448 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.251417    4898 generic.go:356] "Generic (PLEG): container finished" podID="e2da54f0-7885-4cd9-969c-c6051ba28f41" containerID="712c9b8d863f0b8273a953ef20839db3121d72e3f3ff89833ea56cce1b419958" exitCode=60
Nov 02 08:15:18.251555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.251498    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48" event={"ID":"e2da54f0-7885-4cd9-969c-c6051ba28f41","Type":"ContainerDied","Data":"712c9b8d863f0b8273a953ef20839db3121d72e3f3ff89833ea56cce1b419958"}
Nov 02 08:15:18.251555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.251542    4898 scope.go:122] "RemoveContainer" containerID="3bd1c594785c5ef21aa5fd6906e6faa69abf06b9fca4420f02b0508c23103180"
Nov 02 08:15:18.252081 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.252062    4898 scope.go:122] "RemoveContainer" containerID="712c9b8d863f0b8273a953ef20839db3121d72e3f3ff89833ea56cce1b419958"
Nov 02 08:15:18.252514 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:18.252489    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48_odh-ai-gateway-infra(e2da54f0-7885-4cd9-969c-c6051ba28f41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41"
Nov 02 08:15:18.255438 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.255415    4898 generic.go:356] "Generic (PLEG): container finished" podID="ee823ad5-4625-4083-9cc9-20044043dedf" containerID="1d3c3edbb4c6b89eca92fd839e314dda1d8d973dbbfaa6d4bfbefae890bb8389" exitCode=60
Nov 02 08:15:18.255531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.255513    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88" event={"ID":"ee823ad5-4625-4083-9cc9-20044043dedf","Type":"ContainerDied","Data":"1d3c3edbb4c6b89eca92fd839e314dda1d8d973dbbfaa6d4bfbefae890bb8389"}
Nov 02 08:15:18.255956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.255872    4898 scope.go:122] "RemoveContainer" containerID="1d3c3edbb4c6b89eca92fd839e314dda1d8d973dbbfaa6d4bfbefae890bb8389"
Nov 02 08:15:18.256976 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:18.256942    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88_odh-ai-gateway-infra(ee823ad5-4625-4083-9cc9-20044043dedf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88" podUID="ee823ad5-4625-4083-9cc9-20044043dedf"
Nov 02 08:15:18.261722 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.261697    4898 generic.go:356] "Generic (PLEG): container finished" podID="fab5290b-4dd7-494c-9381-a467e78f96c8" containerID="7326b0b703ede0521d3cdb213851c9fda6df926350ec527ae9edbf227f160efa" exitCode=60
Nov 02 08:15:18.261821 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.261805    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq" event={"ID":"fab5290b-4dd7-494c-9381-a467e78f96c8","Type":"ContainerDied","Data":"7326b0b703ede0521d3cdb213851c9fda6df926350ec527ae9edbf227f160efa"}
Nov 02 08:15:18.262246 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.262229    4898 scope.go:122] "RemoveContainer" containerID="7326b0b703ede0521d3cdb213851c9fda6df926350ec527ae9edbf227f160efa"
Nov 02 08:15:18.262570 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:18.262547    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq_odh-ai-gateway-infra(fab5290b-4dd7-494c-9381-a467e78f96c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8"
Nov 02 08:15:18.265127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.264765    4898 generic.go:356] "Generic (PLEG): container finished" podID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" containerID="8bd327064cad6768a1ea84e88c319b4c171ca59d96d14d3603fa6acb1cf4d337" exitCode=60
Nov 02 08:15:18.265127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.264976    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6" event={"ID":"9584c1c3-65cf-4e75-aa0f-9a3681010cc2","Type":"ContainerDied","Data":"8bd327064cad6768a1ea84e88c319b4c171ca59d96d14d3603fa6acb1cf4d337"}
Nov 02 08:15:18.265396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.265374    4898 scope.go:122] "RemoveContainer" containerID="8bd327064cad6768a1ea84e88c319b4c171ca59d96d14d3603fa6acb1cf4d337"
Nov 02 08:15:18.265776 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:18.265747    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6_odh-ai-gateway-infra(9584c1c3-65cf-4e75-aa0f-9a3681010cc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2"
Nov 02 08:15:18.267148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.267121    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv" event={"ID":"1f648095-5449-42df-885a-29e8044cb8c2","Type":"ContainerStarted","Data":"adb52d2711e2f78b0ad625f1997d5d5c7acf5249a3401c492578a09626d7b7f7"}
Nov 02 08:15:18.270326 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.270301    4898 generic.go:356] "Generic (PLEG): container finished" podID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" containerID="83862a56ed23f56f90e704f6befc2a083c8d60484cf10b6976234015cf3ee910" exitCode=60
Nov 02 08:15:18.270523 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.270502    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz" event={"ID":"5ff46f33-1e5a-4a41-85a1-7d18107f3844","Type":"ContainerDied","Data":"83862a56ed23f56f90e704f6befc2a083c8d60484cf10b6976234015cf3ee910"}
Nov 02 08:15:18.271115 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.271093    4898 scope.go:122] "RemoveContainer" containerID="83862a56ed23f56f90e704f6befc2a083c8d60484cf10b6976234015cf3ee910"
Nov 02 08:15:18.271585 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:18.271560    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz_odh-ai-gateway-infra(5ff46f33-1e5a-4a41-85a1-7d18107f3844)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844"
Nov 02 08:15:18.274546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.274491    4898 generic.go:356] "Generic (PLEG): container finished" podID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" containerID="0129697342fd9cc97afc1942c3499b669ba566503a50108665cc8fe58e77c079" exitCode=60
Nov 02 08:15:18.274818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.274558    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s" event={"ID":"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5","Type":"ContainerDied","Data":"0129697342fd9cc97afc1942c3499b669ba566503a50108665cc8fe58e77c079"}
Nov 02 08:15:18.275261 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.275221    4898 scope.go:122] "RemoveContainer" containerID="0129697342fd9cc97afc1942c3499b669ba566503a50108665cc8fe58e77c079"
Nov 02 08:15:18.275572 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:18.275547    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s_odh-ai-gateway-infra(abf3abb8-f4bd-4770-8ce7-5df5defd7fb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5"
Nov 02 08:15:18.277619 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.277601    4898 generic.go:356] "Generic (PLEG): container finished" podID="c81cf60f-13a6-4260-9364-114e89edeef7" containerID="650ba9db60cd35b980155ace4d6e68c4c10a1176e2ac8dddbbe483e5f2b78bc2" exitCode=60
Nov 02 08:15:18.277741 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.277724    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw" event={"ID":"c81cf60f-13a6-4260-9364-114e89edeef7","Type":"ContainerDied","Data":"650ba9db60cd35b980155ace4d6e68c4c10a1176e2ac8dddbbe483e5f2b78bc2"}
Nov 02 08:15:18.278186 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.278170    4898 scope.go:122] "RemoveContainer" containerID="650ba9db60cd35b980155ace4d6e68c4c10a1176e2ac8dddbbe483e5f2b78bc2"
Nov 02 08:15:18.278439 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:18.278415    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw_odh-ai-gateway-infra(c81cf60f-13a6-4260-9364-114e89edeef7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw" podUID="c81cf60f-13a6-4260-9364-114e89edeef7"
Nov 02 08:15:18.279158 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.279132    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6" event={"ID":"6ff85201-d3aa-4dcc-ae86-2a7865b44d82","Type":"ContainerStarted","Data":"bc3e8bb4321913b603978592cda3116eb6a6867037f2c750f5e44d1961a27f19"}
Nov 02 08:15:18.281058 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.280993    4898 generic.go:356] "Generic (PLEG): container finished" podID="a2876d3b-6372-4dd1-919d-4a53af9540bb" containerID="29023b844e4de8b24827b494a9aefb595d64ef4ba9d326543c3df8a60dc9ad24" exitCode=60
Nov 02 08:15:18.281217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.281062    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l" event={"ID":"a2876d3b-6372-4dd1-919d-4a53af9540bb","Type":"ContainerDied","Data":"29023b844e4de8b24827b494a9aefb595d64ef4ba9d326543c3df8a60dc9ad24"}
Nov 02 08:15:18.282982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.282957    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7" event={"ID":"7ebd6238-df9e-4a53-8f56-66b32a3167fd","Type":"ContainerStarted","Data":"929bcf7b2572be617be64edaa20c6fd01ac59c29cc8bd255fe906eb7564cbcd0"}
Nov 02 08:15:18.283234 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.283222    4898 scope.go:122] "RemoveContainer" containerID="29023b844e4de8b24827b494a9aefb595d64ef4ba9d326543c3df8a60dc9ad24"
Nov 02 08:15:18.283617 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:18.283594    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l_odh-ai-gateway-infra(a2876d3b-6372-4dd1-919d-4a53af9540bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb"
Nov 02 08:15:18.307164 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.307114    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr" podStartSLOduration=2.307098991 podStartE2EDuration="2.307098991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:18.292187857 +0000 UTC m=+3068.982514841" watchObservedRunningTime="2026-11-02 08:15:18.307098991 +0000 UTC m=+3068.997425993"
Nov 02 08:15:18.414356 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.414328    4898 scope.go:122] "RemoveContainer" containerID="7c40d8f1825968d4ddaf369cb71f46ce504ab6ff36e2362bed5f70f8a37fbdbb"
Nov 02 08:15:18.430650 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.430622    4898 scope.go:122] "RemoveContainer" containerID="2af11f40f2476b3614447ea5f580189cc5e26d3de23ec6a12b342fcde618990a"
Nov 02 08:15:18.447069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.447043    4898 scope.go:122] "RemoveContainer" containerID="d3c9d94a1cd8a047ebb40ed734f8029118dad50ff6aff3250e7e9a29ada87b5f"
Nov 02 08:15:18.459904 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.459865    4898 scope.go:122] "RemoveContainer" containerID="97cad9c069e0106eec36542775f9d81c605a774ec62e154dd8131a4f6d375d39"
Nov 02 08:15:18.462556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.460053    4898 scope.go:122] "RemoveContainer" containerID="d559ed4b90b4fcf569b2a38aa7c3ecc3d2ebf61d10ce98bad3148c41668ab73b"
Nov 02 08:15:18.462556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.460622    4898 scope.go:122] "RemoveContainer" containerID="8109cab730c32f848447f5f3d21d601cc8615c8d619fffc4fb801b2a283c0bb8"
Nov 02 08:15:18.477184 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.475487    4898 scope.go:122] "RemoveContainer" containerID="18f9173ea272240ed1bdcfb60555a5e221f08c491d89d449a8290b2fcd44ae06"
Nov 02 08:15:18.532596 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.532444    4898 scope.go:122] "RemoveContainer" containerID="b9835992b195fe924a36266980d50468e3af07516fa0ddaf46cf388966b6e7ed"
Nov 02 08:15:18.575949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.574442    4898 scope.go:122] "RemoveContainer" containerID="4bb56aae802847eac7f75085c548acca37e3c98bf237fc74a184b90ad1ab0580"
Nov 02 08:15:18.604272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:18.604233    4898 scope.go:122] "RemoveContainer" containerID="7a0de784629fa01953e43776b785f65f68ff68b35a72a9469750ee79e57d5459"
Nov 02 08:15:19.306399 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.306356    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6" event={"ID":"6ff85201-d3aa-4dcc-ae86-2a7865b44d82","Type":"ContainerStarted","Data":"82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46"}
Nov 02 08:15:19.306752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.306721    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:19.308883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.308853    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq" event={"ID":"c3200858-4b8b-4500-bc2f-8fa185f752f6","Type":"ContainerStarted","Data":"3bf55c42e1b8648740f3a06ccbeda8f2e362017b9fc3b9566eaf394c49df6456"}
Nov 02 08:15:19.313049 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.313023    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8" event={"ID":"7bca6afb-be58-48e6-8d4f-317c45edcffd","Type":"ContainerStarted","Data":"5d6c792e540d9f57ff18419a81d1dd994e20f1687ae093b3150f0aef328a985e"}
Nov 02 08:15:19.315087 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.315058    4898 generic.go:356] "Generic (PLEG): container finished" podID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" containerID="929bcf7b2572be617be64edaa20c6fd01ac59c29cc8bd255fe906eb7564cbcd0" exitCode=60
Nov 02 08:15:19.315199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.315099    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7" event={"ID":"7ebd6238-df9e-4a53-8f56-66b32a3167fd","Type":"ContainerDied","Data":"929bcf7b2572be617be64edaa20c6fd01ac59c29cc8bd255fe906eb7564cbcd0"}
Nov 02 08:15:19.315199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.315161    4898 scope.go:122] "RemoveContainer" containerID="08b479e355f416c341789e5f62fc727cabb38c32b1b6565818576ff514caf3ee"
Nov 02 08:15:19.315524 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.315504    4898 scope.go:122] "RemoveContainer" containerID="929bcf7b2572be617be64edaa20c6fd01ac59c29cc8bd255fe906eb7564cbcd0"
Nov 02 08:15:19.316231 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:19.316062    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7_odh-ai-gateway-infra(7ebd6238-df9e-4a53-8f56-66b32a3167fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd"
Nov 02 08:15:19.326600 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.326575    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff" event={"ID":"0615ef0f-a1d8-47cc-a6b4-481fadc306bc","Type":"ContainerStarted","Data":"9a2bc8fb1efe474cafa237834a33bc8b9f9d956fac29b3a335db378653eb07c7"}
Nov 02 08:15:19.328642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.328615    4898 generic.go:356] "Generic (PLEG): container finished" podID="1f648095-5449-42df-885a-29e8044cb8c2" containerID="adb52d2711e2f78b0ad625f1997d5d5c7acf5249a3401c492578a09626d7b7f7" exitCode=60
Nov 02 08:15:19.328762 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.328677    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv" event={"ID":"1f648095-5449-42df-885a-29e8044cb8c2","Type":"ContainerDied","Data":"adb52d2711e2f78b0ad625f1997d5d5c7acf5249a3401c492578a09626d7b7f7"}
Nov 02 08:15:19.329135 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.329112    4898 scope.go:122] "RemoveContainer" containerID="adb52d2711e2f78b0ad625f1997d5d5c7acf5249a3401c492578a09626d7b7f7"
Nov 02 08:15:19.329437 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:19.329413    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv_odh-ai-gateway-infra(1f648095-5449-42df-885a-29e8044cb8c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv" podUID="1f648095-5449-42df-885a-29e8044cb8c2"
Nov 02 08:15:19.332536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.332508    4898 scope.go:122] "RemoveContainer" containerID="8fd44eb5808145b94b4867fe641a9f8688b4a9e0ae70903069b5befc3ea90734"
Nov 02 08:15:19.345423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.345396    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"]
Nov 02 08:15:19.359663 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.359604    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6" podStartSLOduration=3.359587599 podStartE2EDuration="3.359587599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:19.348657669 +0000 UTC m=+3070.038984655" watchObservedRunningTime="2026-11-02 08:15:19.359587599 +0000 UTC m=+3070.049914563"
Nov 02 08:15:19.383511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.383477    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"]
Nov 02 08:15:19.425157 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.425129    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"]
Nov 02 08:15:19.515229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.515192    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"]
Nov 02 08:15:19.517100 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.516773    4898 scope.go:122] "RemoveContainer" containerID="8a5f9b664d5cebc5bc2259d5986151db0a410fcd3290856016fcfdc1cced9a27"
Nov 02 08:15:19.518163 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.517519    4898 scope.go:122] "RemoveContainer" containerID="f5c264d77e8b72e454a1e404a3e460276b5bcf6d81fefa60366604b3223777e5"
Nov 02 08:15:19.518970 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.518569    4898 scope.go:122] "RemoveContainer" containerID="f7b521eb30f7e43e99cd2100de19eff81864f8be04bd8965ac30afda4d41ff3f"
Nov 02 08:15:19.536750 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.533850    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"]
Nov 02 08:15:19.573434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.564255    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"]
Nov 02 08:15:19.638009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.637862    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"]
Nov 02 08:15:19.666103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.665877    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"]
Nov 02 08:15:19.691993 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.691908    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"]
Nov 02 08:15:19.735594 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.735255    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"]
Nov 02 08:15:19.743014 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.742645    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"
Nov 02 08:15:19.801270 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.801238    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"
Nov 02 08:15:19.830618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.830578    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff46f33-1e5a-4a41-85a1-7d18107f3844-kube-api-access-84784\" (UniqueName: \"kubernetes.io/projected/5ff46f33-1e5a-4a41-85a1-7d18107f3844-kube-api-access-84784\") pod \"5ff46f33-1e5a-4a41-85a1-7d18107f3844\" (UID: \"5ff46f33-1e5a-4a41-85a1-7d18107f3844\") "
Nov 02 08:15:19.830770 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.830684    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff46f33-1e5a-4a41-85a1-7d18107f3844-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff46f33-1e5a-4a41-85a1-7d18107f3844-maas-api-service-ca\") pod \"5ff46f33-1e5a-4a41-85a1-7d18107f3844\" (UID: \"5ff46f33-1e5a-4a41-85a1-7d18107f3844\") "
Nov 02 08:15:19.835793 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.835743    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff46f33-1e5a-4a41-85a1-7d18107f3844-maas-api-service-ca" pod "5ff46f33-1e5a-4a41-85a1-7d18107f3844" (UID: "5ff46f33-1e5a-4a41-85a1-7d18107f3844"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:19.836563 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.836527    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff46f33-1e5a-4a41-85a1-7d18107f3844-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:19.850106 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:19.850055    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff46f33-1e5a-4a41-85a1-7d18107f3844-kube-api-access-84784" pod "5ff46f33-1e5a-4a41-85a1-7d18107f3844" (UID: "5ff46f33-1e5a-4a41-85a1-7d18107f3844"). InnerVolumeSpecName "kube-api-access-84784". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:20.067005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.012726    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee823ad5-4625-4083-9cc9-20044043dedf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee823ad5-4625-4083-9cc9-20044043dedf-maas-api-service-ca\") pod \"ee823ad5-4625-4083-9cc9-20044043dedf\" (UID: \"ee823ad5-4625-4083-9cc9-20044043dedf\") "
Nov 02 08:15:20.067005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.012928    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee823ad5-4625-4083-9cc9-20044043dedf-kube-api-access-qvgm7\" (UniqueName: \"kubernetes.io/projected/ee823ad5-4625-4083-9cc9-20044043dedf-kube-api-access-qvgm7\") pod \"ee823ad5-4625-4083-9cc9-20044043dedf\" (UID: \"ee823ad5-4625-4083-9cc9-20044043dedf\") "
Nov 02 08:15:20.067005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.014355    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84784\" (UniqueName: \"kubernetes.io/projected/5ff46f33-1e5a-4a41-85a1-7d18107f3844-kube-api-access-84784\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.067005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.029542    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee823ad5-4625-4083-9cc9-20044043dedf-maas-api-service-ca" pod "ee823ad5-4625-4083-9cc9-20044043dedf" (UID: "ee823ad5-4625-4083-9cc9-20044043dedf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:20.067005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.057362    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee823ad5-4625-4083-9cc9-20044043dedf-kube-api-access-qvgm7" pod "ee823ad5-4625-4083-9cc9-20044043dedf" (UID: "ee823ad5-4625-4083-9cc9-20044043dedf"). InnerVolumeSpecName "kube-api-access-qvgm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:20.119152 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.117721    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee823ad5-4625-4083-9cc9-20044043dedf-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.119152 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.117763    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvgm7\" (UniqueName: \"kubernetes.io/projected/ee823ad5-4625-4083-9cc9-20044043dedf-kube-api-access-qvgm7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.199736 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.199704    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"
Nov 02 08:15:20.291950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.291668    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"
Nov 02 08:15:20.311000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.310777    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"
Nov 02 08:15:20.323263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.321365    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-maas-api-service-ca\") pod \"9584c1c3-65cf-4e75-aa0f-9a3681010cc2\" (UID: \"9584c1c3-65cf-4e75-aa0f-9a3681010cc2\") "
Nov 02 08:15:20.323263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.321573    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-kube-api-access-vqvh4\" (UniqueName: \"kubernetes.io/projected/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-kube-api-access-vqvh4\") pod \"9584c1c3-65cf-4e75-aa0f-9a3681010cc2\" (UID: \"9584c1c3-65cf-4e75-aa0f-9a3681010cc2\") "
Nov 02 08:15:20.323473 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.323406    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-maas-api-service-ca" pod "9584c1c3-65cf-4e75-aa0f-9a3681010cc2" (UID: "9584c1c3-65cf-4e75-aa0f-9a3681010cc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:20.325615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.325574    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-kube-api-access-vqvh4" pod "9584c1c3-65cf-4e75-aa0f-9a3681010cc2" (UID: "9584c1c3-65cf-4e75-aa0f-9a3681010cc2"). InnerVolumeSpecName "kube-api-access-vqvh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:20.359056 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.345295    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"
Nov 02 08:15:20.364541 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.364509    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"
Nov 02 08:15:20.371108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.368872    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc" event={"ID":"d8186095-8ca2-42ab-aedf-d0d44811258a","Type":"ContainerStarted","Data":"8d03c8deedf971edd4181893c7bc81de50bfaa971d37af2b895bb6398dcc38f1"}
Nov 02 08:15:20.374429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.373289    4898 generic.go:356] "Generic (PLEG): container finished" podID="c3200858-4b8b-4500-bc2f-8fa185f752f6" containerID="3bf55c42e1b8648740f3a06ccbeda8f2e362017b9fc3b9566eaf394c49df6456" exitCode=60
Nov 02 08:15:20.374429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.373380    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq" event={"ID":"c3200858-4b8b-4500-bc2f-8fa185f752f6","Type":"ContainerDied","Data":"3bf55c42e1b8648740f3a06ccbeda8f2e362017b9fc3b9566eaf394c49df6456"}
Nov 02 08:15:20.374429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.373427    4898 scope.go:122] "RemoveContainer" containerID="97cad9c069e0106eec36542775f9d81c605a774ec62e154dd8131a4f6d375d39"
Nov 02 08:15:20.374429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.373914    4898 scope.go:122] "RemoveContainer" containerID="3bf55c42e1b8648740f3a06ccbeda8f2e362017b9fc3b9566eaf394c49df6456"
Nov 02 08:15:20.374429 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:20.374381    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq_odh-ai-gateway-infra(c3200858-4b8b-4500-bc2f-8fa185f752f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6"
Nov 02 08:15:20.386988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.382148    4898 generic.go:356] "Generic (PLEG): container finished" podID="7bca6afb-be58-48e6-8d4f-317c45edcffd" containerID="5d6c792e540d9f57ff18419a81d1dd994e20f1687ae093b3150f0aef328a985e" exitCode=60
Nov 02 08:15:20.386988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.382230    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8" event={"ID":"7bca6afb-be58-48e6-8d4f-317c45edcffd","Type":"ContainerDied","Data":"5d6c792e540d9f57ff18419a81d1dd994e20f1687ae093b3150f0aef328a985e"}
Nov 02 08:15:20.386988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.382642    4898 scope.go:122] "RemoveContainer" containerID="5d6c792e540d9f57ff18419a81d1dd994e20f1687ae093b3150f0aef328a985e"
Nov 02 08:15:20.386988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.383086    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"
Nov 02 08:15:20.388918 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:20.388179    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8_odh-ai-gateway-infra(7bca6afb-be58-48e6-8d4f-317c45edcffd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd"
Nov 02 08:15:20.411757 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.411711    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48" event={"ID":"e2da54f0-7885-4cd9-969c-c6051ba28f41","Type":"ContainerDied","Data":"90bbcc1bf29a4c045b5c7b573e848aed89de897258000d3f4ead9d31274b97ed"}
Nov 02 08:15:20.411953 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.411729    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"
Nov 02 08:15:20.416703 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.414478    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88" event={"ID":"ee823ad5-4625-4083-9cc9-20044043dedf","Type":"ContainerDied","Data":"8dcf327bb0f64943788a99097911f2384050676d096592a57b81f7ac76620a05"}
Nov 02 08:15:20.416703 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.414605    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"
Nov 02 08:15:20.422656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.420455    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq" event={"ID":"fab5290b-4dd7-494c-9381-a467e78f96c8","Type":"ContainerDied","Data":"c3071f78cdd272038646ee4d90b6f7a90824b7e95788927e64a6f86f4dc207a6"}
Nov 02 08:15:20.422656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.420560    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"
Nov 02 08:15:20.422656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.422239    4898 scope.go:122] "RemoveContainer" containerID="8109cab730c32f848447f5f3d21d601cc8615c8d619fffc4fb801b2a283c0bb8"
Nov 02 08:15:20.425957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.423410    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-maas-api-service-ca\") pod \"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5\" (UID: \"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5\") "
Nov 02 08:15:20.425957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.423545    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81cf60f-13a6-4260-9364-114e89edeef7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81cf60f-13a6-4260-9364-114e89edeef7-maas-api-service-ca\") pod \"c81cf60f-13a6-4260-9364-114e89edeef7\" (UID: \"c81cf60f-13a6-4260-9364-114e89edeef7\") "
Nov 02 08:15:20.425957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.423590    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81cf60f-13a6-4260-9364-114e89edeef7-kube-api-access-hdds4\" (UniqueName: \"kubernetes.io/projected/c81cf60f-13a6-4260-9364-114e89edeef7-kube-api-access-hdds4\") pod \"c81cf60f-13a6-4260-9364-114e89edeef7\" (UID: \"c81cf60f-13a6-4260-9364-114e89edeef7\") "
Nov 02 08:15:20.425957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.423686    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab5290b-4dd7-494c-9381-a467e78f96c8-kube-api-access-z7qpw\" (UniqueName: \"kubernetes.io/projected/fab5290b-4dd7-494c-9381-a467e78f96c8-kube-api-access-z7qpw\") pod \"fab5290b-4dd7-494c-9381-a467e78f96c8\" (UID: \"fab5290b-4dd7-494c-9381-a467e78f96c8\") "
Nov 02 08:15:20.425957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.423722    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-kube-api-access-hf2pb\" (UniqueName: \"kubernetes.io/projected/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-kube-api-access-hf2pb\") pod \"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5\" (UID: \"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5\") "
Nov 02 08:15:20.425957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.423781    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2da54f0-7885-4cd9-969c-c6051ba28f41-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da54f0-7885-4cd9-969c-c6051ba28f41-maas-api-service-ca\") pod \"e2da54f0-7885-4cd9-969c-c6051ba28f41\" (UID: \"e2da54f0-7885-4cd9-969c-c6051ba28f41\") "
Nov 02 08:15:20.425957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.423822    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2da54f0-7885-4cd9-969c-c6051ba28f41-kube-api-access-lzjws\" (UniqueName: \"kubernetes.io/projected/e2da54f0-7885-4cd9-969c-c6051ba28f41-kube-api-access-lzjws\") pod \"e2da54f0-7885-4cd9-969c-c6051ba28f41\" (UID: \"e2da54f0-7885-4cd9-969c-c6051ba28f41\") "
Nov 02 08:15:20.425957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.423944    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab5290b-4dd7-494c-9381-a467e78f96c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab5290b-4dd7-494c-9381-a467e78f96c8-maas-api-service-ca\") pod \"fab5290b-4dd7-494c-9381-a467e78f96c8\" (UID: \"fab5290b-4dd7-494c-9381-a467e78f96c8\") "
Nov 02 08:15:20.425957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.424818    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.425957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.424841    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqvh4\" (UniqueName: \"kubernetes.io/projected/9584c1c3-65cf-4e75-aa0f-9a3681010cc2-kube-api-access-vqvh4\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.425957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.425534    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-maas-api-service-ca" pod "abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" (UID: "abf3abb8-f4bd-4770-8ce7-5df5defd7fb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:20.425957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.425862    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81cf60f-13a6-4260-9364-114e89edeef7-maas-api-service-ca" pod "c81cf60f-13a6-4260-9364-114e89edeef7" (UID: "c81cf60f-13a6-4260-9364-114e89edeef7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:20.430204 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.427721    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2da54f0-7885-4cd9-969c-c6051ba28f41-maas-api-service-ca" pod "e2da54f0-7885-4cd9-969c-c6051ba28f41" (UID: "e2da54f0-7885-4cd9-969c-c6051ba28f41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:20.430204 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.428074    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab5290b-4dd7-494c-9381-a467e78f96c8-maas-api-service-ca" pod "fab5290b-4dd7-494c-9381-a467e78f96c8" (UID: "fab5290b-4dd7-494c-9381-a467e78f96c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:20.430204 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.428673    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81cf60f-13a6-4260-9364-114e89edeef7-kube-api-access-hdds4" pod "c81cf60f-13a6-4260-9364-114e89edeef7" (UID: "c81cf60f-13a6-4260-9364-114e89edeef7"). InnerVolumeSpecName "kube-api-access-hdds4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:20.430204 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.429555    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2da54f0-7885-4cd9-969c-c6051ba28f41-kube-api-access-lzjws" pod "e2da54f0-7885-4cd9-969c-c6051ba28f41" (UID: "e2da54f0-7885-4cd9-969c-c6051ba28f41"). InnerVolumeSpecName "kube-api-access-lzjws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:20.430204 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.429712    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"
Nov 02 08:15:20.430204 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.429757    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6" event={"ID":"9584c1c3-65cf-4e75-aa0f-9a3681010cc2","Type":"ContainerDied","Data":"05277329815a0d6d4605abdc3005738b3fa06c84c2e5af56e391138ec8e9d3d8"}
Nov 02 08:15:20.431022 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.430937    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab5290b-4dd7-494c-9381-a467e78f96c8-kube-api-access-z7qpw" pod "fab5290b-4dd7-494c-9381-a467e78f96c8" (UID: "fab5290b-4dd7-494c-9381-a467e78f96c8"). InnerVolumeSpecName "kube-api-access-z7qpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:20.434037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.434009    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-kube-api-access-hf2pb" pod "abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" (UID: "abf3abb8-f4bd-4770-8ce7-5df5defd7fb5"). InnerVolumeSpecName "kube-api-access-hf2pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:20.434399 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.434370    4898 generic.go:356] "Generic (PLEG): container finished" podID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" containerID="9a2bc8fb1efe474cafa237834a33bc8b9f9d956fac29b3a335db378653eb07c7" exitCode=60
Nov 02 08:15:20.434570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.434549    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff" event={"ID":"0615ef0f-a1d8-47cc-a6b4-481fadc306bc","Type":"ContainerDied","Data":"9a2bc8fb1efe474cafa237834a33bc8b9f9d956fac29b3a335db378653eb07c7"}
Nov 02 08:15:20.451153 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.450413    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz" event={"ID":"5ff46f33-1e5a-4a41-85a1-7d18107f3844","Type":"ContainerDied","Data":"305e921db7870511f08920ea9c08bad04ad11dc27975656daf5a938b7b082bb4"}
Nov 02 08:15:20.451153 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.450539    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"
Nov 02 08:15:20.452915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.452642    4898 scope.go:122] "RemoveContainer" containerID="9a2bc8fb1efe474cafa237834a33bc8b9f9d956fac29b3a335db378653eb07c7"
Nov 02 08:15:20.453248 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:20.453195    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff_odh-ai-gateway-infra(0615ef0f-a1d8-47cc-a6b4-481fadc306bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc"
Nov 02 08:15:20.461080 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.459971    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"
Nov 02 08:15:20.461080 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.460875    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s" event={"ID":"abf3abb8-f4bd-4770-8ce7-5df5defd7fb5","Type":"ContainerDied","Data":"f2dbb8adb7a63f5ba4ce1b33ad8537a880d7cb1a73d5fb04475b0617f65759c4"}
Nov 02 08:15:20.462968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.462942    4898 scope.go:122] "RemoveContainer" containerID="488f972f6b1eb8e70830222a250dc62e6d9cafc7ae44ccf3d3f2f1f97c106f63"
Nov 02 08:15:20.468285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.468262    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"
Nov 02 08:15:20.476737 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.476260    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw" event={"ID":"c81cf60f-13a6-4260-9364-114e89edeef7","Type":"ContainerDied","Data":"c958c7a5dc72ad93efa1729138122cc614450f90c2d87dfa7caef59a22a5ebb0"}
Nov 02 08:15:20.476737 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.476440    4898 scope.go:122] "RemoveContainer" containerID="712c9b8d863f0b8273a953ef20839db3121d72e3f3ff89833ea56cce1b419958"
Nov 02 08:15:20.513183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.513149    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"]
Nov 02 08:15:20.536103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.536068    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2876d3b-6372-4dd1-919d-4a53af9540bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2876d3b-6372-4dd1-919d-4a53af9540bb-maas-api-service-ca\") pod \"a2876d3b-6372-4dd1-919d-4a53af9540bb\" (UID: \"a2876d3b-6372-4dd1-919d-4a53af9540bb\") "
Nov 02 08:15:20.536263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.536217    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2876d3b-6372-4dd1-919d-4a53af9540bb-kube-api-access-k6rq6\" (UniqueName: \"kubernetes.io/projected/a2876d3b-6372-4dd1-919d-4a53af9540bb-kube-api-access-k6rq6\") pod \"a2876d3b-6372-4dd1-919d-4a53af9540bb\" (UID: \"a2876d3b-6372-4dd1-919d-4a53af9540bb\") "
Nov 02 08:15:20.536725 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.536696    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2876d3b-6372-4dd1-919d-4a53af9540bb-maas-api-service-ca" pod "a2876d3b-6372-4dd1-919d-4a53af9540bb" (UID: "a2876d3b-6372-4dd1-919d-4a53af9540bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:20.537365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.537339    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.537775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.537370    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c81cf60f-13a6-4260-9364-114e89edeef7-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.537775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.537388    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdds4\" (UniqueName: \"kubernetes.io/projected/c81cf60f-13a6-4260-9364-114e89edeef7-kube-api-access-hdds4\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.537775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.537402    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2876d3b-6372-4dd1-919d-4a53af9540bb-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.537775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.537416    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7qpw\" (UniqueName: \"kubernetes.io/projected/fab5290b-4dd7-494c-9381-a467e78f96c8-kube-api-access-z7qpw\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.537775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.537430    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf2pb\" (UniqueName: \"kubernetes.io/projected/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5-kube-api-access-hf2pb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.537775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.537445    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da54f0-7885-4cd9-969c-c6051ba28f41-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.537775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.537469    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzjws\" (UniqueName: \"kubernetes.io/projected/e2da54f0-7885-4cd9-969c-c6051ba28f41-kube-api-access-lzjws\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.537775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.537483    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fab5290b-4dd7-494c-9381-a467e78f96c8-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.551692 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.551656    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2876d3b-6372-4dd1-919d-4a53af9540bb-kube-api-access-k6rq6" pod "a2876d3b-6372-4dd1-919d-4a53af9540bb" (UID: "a2876d3b-6372-4dd1-919d-4a53af9540bb"). InnerVolumeSpecName "kube-api-access-k6rq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:20.591634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.591546    4898 scope.go:122] "RemoveContainer" containerID="1d3c3edbb4c6b89eca92fd839e314dda1d8d973dbbfaa6d4bfbefae890bb8389"
Nov 02 08:15:20.613923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.613849    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"]
Nov 02 08:15:20.643553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.642062    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6rq6\" (UniqueName: \"kubernetes.io/projected/a2876d3b-6372-4dd1-919d-4a53af9540bb-kube-api-access-k6rq6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.658830 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.658768    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"]
Nov 02 08:15:20.677079 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.677050    4898 scope.go:122] "RemoveContainer" containerID="7326b0b703ede0521d3cdb213851c9fda6df926350ec527ae9edbf227f160efa"
Nov 02 08:15:20.711169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.711125    4898 scope.go:122] "RemoveContainer" containerID="8bd327064cad6768a1ea84e88c319b4c171ca59d96d14d3603fa6acb1cf4d337"
Nov 02 08:15:20.754937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.747463    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"
Nov 02 08:15:20.769176 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.769149    4898 scope.go:122] "RemoveContainer" containerID="d559ed4b90b4fcf569b2a38aa7c3ecc3d2ebf61d10ce98bad3148c41668ab73b"
Nov 02 08:15:20.831964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.831940    4898 scope.go:122] "RemoveContainer" containerID="83862a56ed23f56f90e704f6befc2a083c8d60484cf10b6976234015cf3ee910"
Nov 02 08:15:20.845612 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.845543    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebd6238-df9e-4a53-8f56-66b32a3167fd-kube-api-access-kbh87\" (UniqueName: \"kubernetes.io/projected/7ebd6238-df9e-4a53-8f56-66b32a3167fd-kube-api-access-kbh87\") pod \"7ebd6238-df9e-4a53-8f56-66b32a3167fd\" (UID: \"7ebd6238-df9e-4a53-8f56-66b32a3167fd\") "
Nov 02 08:15:20.845852 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.845829    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ebd6238-df9e-4a53-8f56-66b32a3167fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebd6238-df9e-4a53-8f56-66b32a3167fd-maas-api-service-ca\") pod \"7ebd6238-df9e-4a53-8f56-66b32a3167fd\" (UID: \"7ebd6238-df9e-4a53-8f56-66b32a3167fd\") "
Nov 02 08:15:20.846726 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.846699    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebd6238-df9e-4a53-8f56-66b32a3167fd-maas-api-service-ca" pod "7ebd6238-df9e-4a53-8f56-66b32a3167fd" (UID: "7ebd6238-df9e-4a53-8f56-66b32a3167fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:20.855458 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.855423    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebd6238-df9e-4a53-8f56-66b32a3167fd-kube-api-access-kbh87" pod "7ebd6238-df9e-4a53-8f56-66b32a3167fd" (UID: "7ebd6238-df9e-4a53-8f56-66b32a3167fd"). InnerVolumeSpecName "kube-api-access-kbh87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:20.855604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.855562    4898 scope.go:122] "RemoveContainer" containerID="0129697342fd9cc97afc1942c3499b669ba566503a50108665cc8fe58e77c079"
Nov 02 08:15:20.865205 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.864207    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"
Nov 02 08:15:20.867928 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.867127    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"]
Nov 02 08:15:20.881370 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.875981    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893455-6vm88"]
Nov 02 08:15:20.923976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.921232    4898 scope.go:122] "RemoveContainer" containerID="650ba9db60cd35b980155ace4d6e68c4c10a1176e2ac8dddbbe483e5f2b78bc2"
Nov 02 08:15:20.931938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.931869    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"]
Nov 02 08:15:20.938142 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.938116    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-736f7c-29893455-k4nm6"]
Nov 02 08:15:20.947396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.947362    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f648095-5449-42df-885a-29e8044cb8c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f648095-5449-42df-885a-29e8044cb8c2-maas-api-service-ca\") pod \"1f648095-5449-42df-885a-29e8044cb8c2\" (UID: \"1f648095-5449-42df-885a-29e8044cb8c2\") "
Nov 02 08:15:20.947555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.947466    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f648095-5449-42df-885a-29e8044cb8c2-kube-api-access-7gvjc\" (UniqueName: \"kubernetes.io/projected/1f648095-5449-42df-885a-29e8044cb8c2-kube-api-access-7gvjc\") pod \"1f648095-5449-42df-885a-29e8044cb8c2\" (UID: \"1f648095-5449-42df-885a-29e8044cb8c2\") "
Nov 02 08:15:20.947719 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.947695    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f648095-5449-42df-885a-29e8044cb8c2-maas-api-service-ca" pod "1f648095-5449-42df-885a-29e8044cb8c2" (UID: "1f648095-5449-42df-885a-29e8044cb8c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:20.948338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.948320    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebd6238-df9e-4a53-8f56-66b32a3167fd-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.948401 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.948339    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbh87\" (UniqueName: \"kubernetes.io/projected/7ebd6238-df9e-4a53-8f56-66b32a3167fd-kube-api-access-kbh87\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.948401 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.948348    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f648095-5449-42df-885a-29e8044cb8c2-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:20.949946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.949920    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f648095-5449-42df-885a-29e8044cb8c2-kube-api-access-7gvjc" pod "1f648095-5449-42df-885a-29e8044cb8c2" (UID: "1f648095-5449-42df-885a-29e8044cb8c2"). InnerVolumeSpecName "kube-api-access-7gvjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:20.972801 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.971865    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"]
Nov 02 08:15:20.988931 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:20.988831    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893455-5gqhw"]
Nov 02 08:15:21.015459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.015424    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"]
Nov 02 08:15:21.024003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.023961    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893455-ftfkq"]
Nov 02 08:15:21.049721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.049691    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gvjc\" (UniqueName: \"kubernetes.io/projected/1f648095-5449-42df-885a-29e8044cb8c2-kube-api-access-7gvjc\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:21.155514 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.155463    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"]
Nov 02 08:15:21.163327 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.163283    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893455-nhc48"]
Nov 02 08:15:21.235297 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.235261    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"]
Nov 02 08:15:21.248536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.248495    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2dde84-29893455-67q9s"]
Nov 02 08:15:21.295277 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.295235    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"]
Nov 02 08:15:21.314179 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.314139    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893455-75zjz"]
Nov 02 08:15:21.486075 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.486034    4898 generic.go:356] "Generic (PLEG): container finished" podID="d8186095-8ca2-42ab-aedf-d0d44811258a" containerID="8d03c8deedf971edd4181893c7bc81de50bfaa971d37af2b895bb6398dcc38f1" exitCode=60
Nov 02 08:15:21.508985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.508656    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" path="/var/lib/kubelet/pods/5ff46f33-1e5a-4a41-85a1-7d18107f3844/volumes"
Nov 02 08:15:21.509447 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.509417    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" path="/var/lib/kubelet/pods/9584c1c3-65cf-4e75-aa0f-9a3681010cc2/volumes"
Nov 02 08:15:21.510135 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.510087    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" path="/var/lib/kubelet/pods/abf3abb8-f4bd-4770-8ce7-5df5defd7fb5/volumes"
Nov 02 08:15:21.511992 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.511970    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81cf60f-13a6-4260-9364-114e89edeef7" path="/var/lib/kubelet/pods/c81cf60f-13a6-4260-9364-114e89edeef7/volumes"
Nov 02 08:15:21.512801 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.512762    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41" path="/var/lib/kubelet/pods/e2da54f0-7885-4cd9-969c-c6051ba28f41/volumes"
Nov 02 08:15:21.513743 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.513723    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee823ad5-4625-4083-9cc9-20044043dedf" path="/var/lib/kubelet/pods/ee823ad5-4625-4083-9cc9-20044043dedf/volumes"
Nov 02 08:15:21.514396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.514293    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"
Nov 02 08:15:21.515575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.515039    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8" path="/var/lib/kubelet/pods/fab5290b-4dd7-494c-9381-a467e78f96c8/volumes"
Nov 02 08:15:21.516940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.516917    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc" event={"ID":"d8186095-8ca2-42ab-aedf-d0d44811258a","Type":"ContainerDied","Data":"8d03c8deedf971edd4181893c7bc81de50bfaa971d37af2b895bb6398dcc38f1"}
Nov 02 08:15:21.517167 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.517147    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l" event={"ID":"a2876d3b-6372-4dd1-919d-4a53af9540bb","Type":"ContainerDied","Data":"3a61de2f327b362aeb0549c72e194296224fb641b49cf35ca5c3c36ea62c0413"}
Nov 02 08:15:21.517281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.517268    4898 scope.go:122] "RemoveContainer" containerID="f7b521eb30f7e43e99cd2100de19eff81864f8be04bd8965ac30afda4d41ff3f"
Nov 02 08:15:21.518130 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.518104    4898 scope.go:122] "RemoveContainer" containerID="8d03c8deedf971edd4181893c7bc81de50bfaa971d37af2b895bb6398dcc38f1"
Nov 02 08:15:21.518457 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:21.518430    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc_odh-ai-gateway-infra(d8186095-8ca2-42ab-aedf-d0d44811258a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a"
Nov 02 08:15:21.538789 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.538512    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"
Nov 02 08:15:21.538789 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.538574    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7" event={"ID":"7ebd6238-df9e-4a53-8f56-66b32a3167fd","Type":"ContainerDied","Data":"07a2d598a097b7f0474a61242860e17fd127edd86f0c46b19ae8f436ffec6881"}
Nov 02 08:15:21.548722 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.547564    4898 generic.go:356] "Generic (PLEG): container finished" podID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" containerID="073e56bd6d81fd7ba7fc9c0b6e8103b5f43c3df6e248861f06d94c7a5b6738bd" exitCode=60
Nov 02 08:15:21.548722 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.547598    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz" event={"ID":"a6cfc1ff-3ea1-4845-98c2-d02d836507a2","Type":"ContainerDied","Data":"073e56bd6d81fd7ba7fc9c0b6e8103b5f43c3df6e248861f06d94c7a5b6738bd"}
Nov 02 08:15:21.548722 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.548170    4898 scope.go:122] "RemoveContainer" containerID="073e56bd6d81fd7ba7fc9c0b6e8103b5f43c3df6e248861f06d94c7a5b6738bd"
Nov 02 08:15:21.548722 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:21.548526    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz_odh-ai-gateway-infra(a6cfc1ff-3ea1-4845-98c2-d02d836507a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2"
Nov 02 08:15:21.566798 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.565772    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd" event={"ID":"13ef0f94-811c-4542-aac5-f4ba52c18f27","Type":"ContainerStarted","Data":"d3bbaee5192c13cd84100618d7d9ed3443352ad56c06722f09c3d36ee463d26c"}
Nov 02 08:15:21.579161 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.577707    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv" event={"ID":"1f648095-5449-42df-885a-29e8044cb8c2","Type":"ContainerDied","Data":"e8a620bf46e603198967438d337177feea8228c917b1cd67ff5f28b39ec2a5ed"}
Nov 02 08:15:21.579564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.577843    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"
Nov 02 08:15:21.600121 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.600009    4898 scope.go:122] "RemoveContainer" containerID="29023b844e4de8b24827b494a9aefb595d64ef4ba9d326543c3df8a60dc9ad24"
Nov 02 08:15:21.614145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.613179    4898 generic.go:356] "Generic (PLEG): container finished" podID="d95edc7f-4b4b-449d-870c-156648685af2" containerID="00baf63f1ba7ef24ba315d11d3fc750246c9489afcb27175191f20db158f7651" exitCode=60
Nov 02 08:15:21.614145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.613226    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv" event={"ID":"d95edc7f-4b4b-449d-870c-156648685af2","Type":"ContainerDied","Data":"00baf63f1ba7ef24ba315d11d3fc750246c9489afcb27175191f20db158f7651"}
Nov 02 08:15:21.614145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.613795    4898 scope.go:122] "RemoveContainer" containerID="00baf63f1ba7ef24ba315d11d3fc750246c9489afcb27175191f20db158f7651"
Nov 02 08:15:21.614344 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:21.614285    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv_odh-ai-gateway-infra(d95edc7f-4b4b-449d-870c-156648685af2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv" podUID="d95edc7f-4b4b-449d-870c-156648685af2"
Nov 02 08:15:21.629755 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.622572    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"]
Nov 02 08:15:21.632273 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.632248    4898 scope.go:122] "RemoveContainer" containerID="929bcf7b2572be617be64edaa20c6fd01ac59c29cc8bd255fe906eb7564cbcd0"
Nov 02 08:15:21.650540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.650504    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893455-nptb7"]
Nov 02 08:15:21.673499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.673334    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"]
Nov 02 08:15:21.678453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.678373    4898 scope.go:122] "RemoveContainer" containerID="8a5f9b664d5cebc5bc2259d5986151db0a410fcd3290856016fcfdc1cced9a27"
Nov 02 08:15:21.701863 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.701820    4898 scope.go:122] "RemoveContainer" containerID="adb52d2711e2f78b0ad625f1997d5d5c7acf5249a3401c492578a09626d7b7f7"
Nov 02 08:15:21.722331 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.722299    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"]
Nov 02 08:15:21.727082 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.726948    4898 scope.go:122] "RemoveContainer" containerID="f5c264d77e8b72e454a1e404a3e460276b5bcf6d81fefa60366604b3223777e5"
Nov 02 08:15:21.728067 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.728041    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893455-rxn4l"]
Nov 02 08:15:21.869281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.869243    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"]
Nov 02 08:15:21.883181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.883150    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893455-929wv"]
Nov 02 08:15:21.891213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.891111    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"
Nov 02 08:15:21.917916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.917861    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"
Nov 02 08:15:21.942092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.942065    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"
Nov 02 08:15:21.993362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.993325    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bca6afb-be58-48e6-8d4f-317c45edcffd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bca6afb-be58-48e6-8d4f-317c45edcffd-maas-api-service-ca\") pod \"7bca6afb-be58-48e6-8d4f-317c45edcffd\" (UID: \"7bca6afb-be58-48e6-8d4f-317c45edcffd\") "
Nov 02 08:15:21.993539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.993479    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3200858-4b8b-4500-bc2f-8fa185f752f6-kube-api-access-bnfd9\" (UniqueName: \"kubernetes.io/projected/c3200858-4b8b-4500-bc2f-8fa185f752f6-kube-api-access-bnfd9\") pod \"c3200858-4b8b-4500-bc2f-8fa185f752f6\" (UID: \"c3200858-4b8b-4500-bc2f-8fa185f752f6\") "
Nov 02 08:15:21.993601 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.993562    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bca6afb-be58-48e6-8d4f-317c45edcffd-kube-api-access-nglnw\" (UniqueName: \"kubernetes.io/projected/7bca6afb-be58-48e6-8d4f-317c45edcffd-kube-api-access-nglnw\") pod \"7bca6afb-be58-48e6-8d4f-317c45edcffd\" (UID: \"7bca6afb-be58-48e6-8d4f-317c45edcffd\") "
Nov 02 08:15:21.993683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.993615    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3200858-4b8b-4500-bc2f-8fa185f752f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3200858-4b8b-4500-bc2f-8fa185f752f6-maas-api-service-ca\") pod \"c3200858-4b8b-4500-bc2f-8fa185f752f6\" (UID: \"c3200858-4b8b-4500-bc2f-8fa185f752f6\") "
Nov 02 08:15:21.993780 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.993756    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bca6afb-be58-48e6-8d4f-317c45edcffd-maas-api-service-ca" pod "7bca6afb-be58-48e6-8d4f-317c45edcffd" (UID: "7bca6afb-be58-48e6-8d4f-317c45edcffd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:21.994067 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.994042    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3200858-4b8b-4500-bc2f-8fa185f752f6-maas-api-service-ca" pod "c3200858-4b8b-4500-bc2f-8fa185f752f6" (UID: "c3200858-4b8b-4500-bc2f-8fa185f752f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:21.994680 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.994656    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bca6afb-be58-48e6-8d4f-317c45edcffd-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:21.994781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.994685    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3200858-4b8b-4500-bc2f-8fa185f752f6-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:21.996237 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.996197    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bca6afb-be58-48e6-8d4f-317c45edcffd-kube-api-access-nglnw" pod "7bca6afb-be58-48e6-8d4f-317c45edcffd" (UID: "7bca6afb-be58-48e6-8d4f-317c45edcffd"). InnerVolumeSpecName "kube-api-access-nglnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:21.996356 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:21.996313    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3200858-4b8b-4500-bc2f-8fa185f752f6-kube-api-access-bnfd9" pod "c3200858-4b8b-4500-bc2f-8fa185f752f6" (UID: "c3200858-4b8b-4500-bc2f-8fa185f752f6"). InnerVolumeSpecName "kube-api-access-bnfd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:22.099923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.099867    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-kube-api-access-mldts\" (UniqueName: \"kubernetes.io/projected/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-kube-api-access-mldts\") pod \"0615ef0f-a1d8-47cc-a6b4-481fadc306bc\" (UID: \"0615ef0f-a1d8-47cc-a6b4-481fadc306bc\") "
Nov 02 08:15:22.100319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.100297    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-maas-api-service-ca\") pod \"0615ef0f-a1d8-47cc-a6b4-481fadc306bc\" (UID: \"0615ef0f-a1d8-47cc-a6b4-481fadc306bc\") "
Nov 02 08:15:22.100672 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.100644    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-maas-api-service-ca" pod "0615ef0f-a1d8-47cc-a6b4-481fadc306bc" (UID: "0615ef0f-a1d8-47cc-a6b4-481fadc306bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:22.101464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.101413    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnfd9\" (UniqueName: \"kubernetes.io/projected/c3200858-4b8b-4500-bc2f-8fa185f752f6-kube-api-access-bnfd9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:22.101611 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.101594    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nglnw\" (UniqueName: \"kubernetes.io/projected/7bca6afb-be58-48e6-8d4f-317c45edcffd-kube-api-access-nglnw\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:22.101727 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.101714    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:22.103307 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.103279    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-kube-api-access-mldts" pod "0615ef0f-a1d8-47cc-a6b4-481fadc306bc" (UID: "0615ef0f-a1d8-47cc-a6b4-481fadc306bc"). InnerVolumeSpecName "kube-api-access-mldts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:22.224916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.222572    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mldts\" (UniqueName: \"kubernetes.io/projected/0615ef0f-a1d8-47cc-a6b4-481fadc306bc-kube-api-access-mldts\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:22.660285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.660255    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"
Nov 02 08:15:22.678382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.660833    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq" event={"ID":"c3200858-4b8b-4500-bc2f-8fa185f752f6","Type":"ContainerDied","Data":"5f63face04ec92c8d4fa96bd7b8297569b12a7f4f4fd0e2700a24d7db75e6e4f"}
Nov 02 08:15:22.678996 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.678966    4898 scope.go:122] "RemoveContainer" containerID="3bf55c42e1b8648740f3a06ccbeda8f2e362017b9fc3b9566eaf394c49df6456"
Nov 02 08:15:22.710803 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.710764    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8" event={"ID":"7bca6afb-be58-48e6-8d4f-317c45edcffd","Type":"ContainerDied","Data":"0c26d0173478a031a58a8b8b45c6d7cd578c396891eda10291a024dfa73c5295"}
Nov 02 08:15:22.712061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.711243    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"
Nov 02 08:15:22.730246 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.730207    4898 generic.go:356] "Generic (PLEG): container finished" podID="13ef0f94-811c-4542-aac5-f4ba52c18f27" containerID="d3bbaee5192c13cd84100618d7d9ed3443352ad56c06722f09c3d36ee463d26c" exitCode=60
Nov 02 08:15:22.730637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.730609    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd" event={"ID":"13ef0f94-811c-4542-aac5-f4ba52c18f27","Type":"ContainerDied","Data":"d3bbaee5192c13cd84100618d7d9ed3443352ad56c06722f09c3d36ee463d26c"}
Nov 02 08:15:22.731306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.731280    4898 scope.go:122] "RemoveContainer" containerID="d3bbaee5192c13cd84100618d7d9ed3443352ad56c06722f09c3d36ee463d26c"
Nov 02 08:15:22.731965 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:22.731932    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd_odh-ai-gateway-infra(13ef0f94-811c-4542-aac5-f4ba52c18f27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27"
Nov 02 08:15:22.734627 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.734600    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff" event={"ID":"0615ef0f-a1d8-47cc-a6b4-481fadc306bc","Type":"ContainerDied","Data":"3d2c76d0ed7433d64cfba36db644ba9ee4fa045b18fed7f6f45d803c9cc2bb60"}
Nov 02 08:15:22.752781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.734842    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"
Nov 02 08:15:22.785512 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.785481    4898 scope.go:122] "RemoveContainer" containerID="5d6c792e540d9f57ff18419a81d1dd994e20f1687ae093b3150f0aef328a985e"
Nov 02 08:15:22.790843 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.790693    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"]
Nov 02 08:15:22.796511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.796482    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:22.798591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.798567    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"]
Nov 02 08:15:22.805624 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.805599    4898 scope.go:122] "RemoveContainer" containerID="488f972f6b1eb8e70830222a250dc62e6d9cafc7ae44ccf3d3f2f1f97c106f63"
Nov 02 08:15:22.807118 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.806985    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893455-xr4sq"]
Nov 02 08:15:22.825495 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.825474    4898 scope.go:122] "RemoveContainer" containerID="9a2bc8fb1efe474cafa237834a33bc8b9f9d956fac29b3a335db378653eb07c7"
Nov 02 08:15:22.834073 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.833430    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"]
Nov 02 08:15:22.909599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.909463    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"]
Nov 02 08:15:22.920961 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.920926    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-6929a4-29893455-84rff"]
Nov 02 08:15:22.951417 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.951349    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"]
Nov 02 08:15:22.965579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.962968    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e271b1-29893455-x4jp8"]
Nov 02 08:15:22.967925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:22.967177    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"]
Nov 02 08:15:23.006264 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.005880    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"]
Nov 02 08:15:23.037169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.032432    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"
Nov 02 08:15:23.172970 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.165738    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8186095-8ca2-42ab-aedf-d0d44811258a-kube-api-access-blw8w\" (UniqueName: \"kubernetes.io/projected/d8186095-8ca2-42ab-aedf-d0d44811258a-kube-api-access-blw8w\") pod \"d8186095-8ca2-42ab-aedf-d0d44811258a\" (UID: \"d8186095-8ca2-42ab-aedf-d0d44811258a\") "
Nov 02 08:15:23.208979 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.208928    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8186095-8ca2-42ab-aedf-d0d44811258a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8186095-8ca2-42ab-aedf-d0d44811258a-maas-api-service-ca\") pod \"d8186095-8ca2-42ab-aedf-d0d44811258a\" (UID: \"d8186095-8ca2-42ab-aedf-d0d44811258a\") "
Nov 02 08:15:23.210542 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.210458    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8186095-8ca2-42ab-aedf-d0d44811258a-maas-api-service-ca" pod "d8186095-8ca2-42ab-aedf-d0d44811258a" (UID: "d8186095-8ca2-42ab-aedf-d0d44811258a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:23.224430 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.224397    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8186095-8ca2-42ab-aedf-d0d44811258a-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:23.265193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.265144    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8186095-8ca2-42ab-aedf-d0d44811258a-kube-api-access-blw8w" pod "d8186095-8ca2-42ab-aedf-d0d44811258a" (UID: "d8186095-8ca2-42ab-aedf-d0d44811258a"). InnerVolumeSpecName "kube-api-access-blw8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.340947    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"]
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342275    4898 cpu_manager.go:414] "containerMap: removing container" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342300    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342316    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ee823ad5-4625-4083-9cc9-20044043dedf" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342325    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee823ad5-4625-4083-9cc9-20044043dedf" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342337    4898 cpu_manager.go:414] "containerMap: removing container" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342345    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342358    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ee823ad5-4625-4083-9cc9-20044043dedf" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342366    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee823ad5-4625-4083-9cc9-20044043dedf" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342385    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342392    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342402    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342412    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342423    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342464    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342491    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342499    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342508    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342515    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342535    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342543    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342553    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c81cf60f-13a6-4260-9364-114e89edeef7" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342560    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81cf60f-13a6-4260-9364-114e89edeef7" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342568    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c81cf60f-13a6-4260-9364-114e89edeef7" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342576    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81cf60f-13a6-4260-9364-114e89edeef7" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342594    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342601    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342622    4898 cpu_manager.go:414] "containerMap: removing container" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342628    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342641    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342647    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342660    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342668    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342682    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1f648095-5449-42df-885a-29e8044cb8c2" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342689    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f648095-5449-42df-885a-29e8044cb8c2" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342702    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342708    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342717    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342724    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342735    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342742    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342759    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342766    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342778    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342785    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342800    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342807    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342816    4898 cpu_manager.go:414] "containerMap: removing container" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342822    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342831    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c81cf60f-13a6-4260-9364-114e89edeef7" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342837    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81cf60f-13a6-4260-9364-114e89edeef7" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342850    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342857    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" containerName="cleanup"
Nov 02 08:15:23.343752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342873    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.342881    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346499    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346512    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346534    4898 cpu_manager.go:414] "containerMap: removing container" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346541    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346555    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1f648095-5449-42df-885a-29e8044cb8c2" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346562    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f648095-5449-42df-885a-29e8044cb8c2" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346572    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346578    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346590    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346596    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346627    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346634    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346645    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346652    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346918    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f648095-5449-42df-885a-29e8044cb8c2" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346937    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346946    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346960    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346973    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346984    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.346994    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee823ad5-4625-4083-9cc9-20044043dedf" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347002    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347010    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee823ad5-4625-4083-9cc9-20044043dedf" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347020    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347026    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347031    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347039    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347046    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347051    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347059    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347065    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81cf60f-13a6-4260-9364-114e89edeef7" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347069    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81cf60f-13a6-4260-9364-114e89edeef7" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347077    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347085    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347091    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f648095-5449-42df-885a-29e8044cb8c2" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347097    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347104    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347111    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347117    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347123    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347128    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347134    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347139    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347147    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" containerName="cleanup"
Nov 02 08:15:23.349338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.347155    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a" containerName="cleanup"
Nov 02 08:15:23.351489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.350317    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blw8w\" (UniqueName: \"kubernetes.io/projected/d8186095-8ca2-42ab-aedf-d0d44811258a-kube-api-access-blw8w\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:23.354317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.353667    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:23.355505 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.355466    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"]
Nov 02 08:15:23.372320 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.372279    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:15:23.397582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.397483    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"
Nov 02 08:15:23.454506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.454406    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-maas-api-service-ca\") pod \"a6cfc1ff-3ea1-4845-98c2-d02d836507a2\" (UID: \"a6cfc1ff-3ea1-4845-98c2-d02d836507a2\") "
Nov 02 08:15:23.454712 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.454683    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-kube-api-access-b5wws\" (UniqueName: \"kubernetes.io/projected/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-kube-api-access-b5wws\") pod \"a6cfc1ff-3ea1-4845-98c2-d02d836507a2\" (UID: \"a6cfc1ff-3ea1-4845-98c2-d02d836507a2\") "
Nov 02 08:15:23.455300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.455272    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfc62b0a-kwhk6\" (UID: \"74f504d5-53f9-40ec-bf39-074da613f90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:23.455709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.455677    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-maas-api-service-ca" pod "a6cfc1ff-3ea1-4845-98c2-d02d836507a2" (UID: "a6cfc1ff-3ea1-4845-98c2-d02d836507a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:23.455822 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.455784    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg7gn\" (UniqueName: \"kubernetes.io/projected/74f504d5-53f9-40ec-bf39-074da613f90b-kube-api-access-rg7gn\") pod \"maas-api-revoke-sub-dfc62b0a-kwhk6\" (UID: \"74f504d5-53f9-40ec-bf39-074da613f90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:23.456002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.455984    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfc62b0a-kwhk6\" (UID: \"74f504d5-53f9-40ec-bf39-074da613f90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:23.456323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.456298    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:23.472710 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.472667    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-kube-api-access-b5wws" pod "a6cfc1ff-3ea1-4845-98c2-d02d836507a2" (UID: "a6cfc1ff-3ea1-4845-98c2-d02d836507a2"). InnerVolumeSpecName "kube-api-access-b5wws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:23.474852 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.474812    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" path="/var/lib/kubelet/pods/0615ef0f-a1d8-47cc-a6b4-481fadc306bc/volumes"
Nov 02 08:15:23.475563 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.475538    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f648095-5449-42df-885a-29e8044cb8c2" path="/var/lib/kubelet/pods/1f648095-5449-42df-885a-29e8044cb8c2/volumes"
Nov 02 08:15:23.476351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.476328    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd" path="/var/lib/kubelet/pods/7bca6afb-be58-48e6-8d4f-317c45edcffd/volumes"
Nov 02 08:15:23.477461 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.477441    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ebd6238-df9e-4a53-8f56-66b32a3167fd" path="/var/lib/kubelet/pods/7ebd6238-df9e-4a53-8f56-66b32a3167fd/volumes"
Nov 02 08:15:23.485013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.484990    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb" path="/var/lib/kubelet/pods/a2876d3b-6372-4dd1-919d-4a53af9540bb/volumes"
Nov 02 08:15:23.507571 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.493677    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3200858-4b8b-4500-bc2f-8fa185f752f6" path="/var/lib/kubelet/pods/c3200858-4b8b-4500-bc2f-8fa185f752f6/volumes"
Nov 02 08:15:23.707971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.671400    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfc62b0a-kwhk6\" (UID: \"74f504d5-53f9-40ec-bf39-074da613f90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:23.707971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.671574    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg7gn\" (UniqueName: \"kubernetes.io/projected/74f504d5-53f9-40ec-bf39-074da613f90b-kube-api-access-rg7gn\") pod \"maas-api-revoke-sub-dfc62b0a-kwhk6\" (UID: \"74f504d5-53f9-40ec-bf39-074da613f90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:23.707971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.671643    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfc62b0a-kwhk6\" (UID: \"74f504d5-53f9-40ec-bf39-074da613f90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:23.723746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.723709    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfc62b0a-kwhk6\" (UID: \"74f504d5-53f9-40ec-bf39-074da613f90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:23.756072 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.751298    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5wws\" (UniqueName: \"kubernetes.io/projected/a6cfc1ff-3ea1-4845-98c2-d02d836507a2-kube-api-access-b5wws\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:23.789388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.789348    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg7gn\" (UniqueName: \"kubernetes.io/projected/74f504d5-53f9-40ec-bf39-074da613f90b-kube-api-access-rg7gn\") pod \"maas-api-revoke-sub-dfc62b0a-kwhk6\" (UID: \"74f504d5-53f9-40ec-bf39-074da613f90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:23.832479 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.832423    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"]
Nov 02 08:15:23.839629 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.839591    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:23.839811 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.839588    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfc62b0a-kwhk6\" (UID: \"74f504d5-53f9-40ec-bf39-074da613f90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:23.843589 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.843563    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"
Nov 02 08:15:23.848797 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.848780    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"
Nov 02 08:15:23.848981 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.848955    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv" event={"ID":"d95edc7f-4b4b-449d-870c-156648685af2","Type":"ContainerDied","Data":"c95c1407b399908eecbc2fb096c7b506dfc2171da08a8f0fadaa891c53f4b29b"}
Nov 02 08:15:23.849081 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.849015    4898 scope.go:122] "RemoveContainer" containerID="00baf63f1ba7ef24ba315d11d3fc750246c9489afcb27175191f20db158f7651"
Nov 02 08:15:23.851623 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.851601    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc" event={"ID":"d8186095-8ca2-42ab-aedf-d0d44811258a","Type":"ContainerDied","Data":"7d74125d1d5f8e3937be14665dd802cc64aab150eca67de056e98fb04a5e5706"}
Nov 02 08:15:23.852544 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.851613    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"
Nov 02 08:15:23.856391 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.856364    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz" event={"ID":"a6cfc1ff-3ea1-4845-98c2-d02d836507a2","Type":"ContainerDied","Data":"9366a10d2d4607d1c55c445e7f3b52062c93ef39d815e9cf66995ed6493cb5d0"}
Nov 02 08:15:23.856487 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.856465    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"
Nov 02 08:15:23.866349 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.866327    4898 scope.go:122] "RemoveContainer" containerID="8d03c8deedf971edd4181893c7bc81de50bfaa971d37af2b895bb6398dcc38f1"
Nov 02 08:15:23.884280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.884250    4898 scope.go:122] "RemoveContainer" containerID="073e56bd6d81fd7ba7fc9c0b6e8103b5f43c3df6e248861f06d94c7a5b6738bd"
Nov 02 08:15:23.959878 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.959789    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95edc7f-4b4b-449d-870c-156648685af2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95edc7f-4b4b-449d-870c-156648685af2-maas-api-service-ca\") pod \"d95edc7f-4b4b-449d-870c-156648685af2\" (UID: \"d95edc7f-4b4b-449d-870c-156648685af2\") "
Nov 02 08:15:23.960255 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.960232    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95edc7f-4b4b-449d-870c-156648685af2-kube-api-access-jlklk\" (UniqueName: \"kubernetes.io/projected/d95edc7f-4b4b-449d-870c-156648685af2-kube-api-access-jlklk\") pod \"d95edc7f-4b4b-449d-870c-156648685af2\" (UID: \"d95edc7f-4b4b-449d-870c-156648685af2\") "
Nov 02 08:15:23.965997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.965969    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"]
Nov 02 08:15:23.969183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.969158    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893455-qkbvz"]
Nov 02 08:15:23.970300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.970273    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95edc7f-4b4b-449d-870c-156648685af2-maas-api-service-ca" pod "d95edc7f-4b4b-449d-870c-156648685af2" (UID: "d95edc7f-4b4b-449d-870c-156648685af2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:23.974751 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:23.974706    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95edc7f-4b4b-449d-870c-156648685af2-kube-api-access-jlklk" pod "d95edc7f-4b4b-449d-870c-156648685af2" (UID: "d95edc7f-4b4b-449d-870c-156648685af2"). InnerVolumeSpecName "kube-api-access-jlklk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:24.017079 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.017047    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:24.072962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.071718    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95edc7f-4b4b-449d-870c-156648685af2-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:24.072962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.071754    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlklk\" (UniqueName: \"kubernetes.io/projected/d95edc7f-4b4b-449d-870c-156648685af2-kube-api-access-jlklk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:24.082303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.082272    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"
Nov 02 08:15:24.112687 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.112662    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"]
Nov 02 08:15:24.180605 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.180540    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ef0f94-811c-4542-aac5-f4ba52c18f27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ef0f94-811c-4542-aac5-f4ba52c18f27-maas-api-service-ca\") pod \"13ef0f94-811c-4542-aac5-f4ba52c18f27\" (UID: \"13ef0f94-811c-4542-aac5-f4ba52c18f27\") "
Nov 02 08:15:24.180864 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.180833    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ef0f94-811c-4542-aac5-f4ba52c18f27-kube-api-access-c9dxc\" (UniqueName: \"kubernetes.io/projected/13ef0f94-811c-4542-aac5-f4ba52c18f27-kube-api-access-c9dxc\") pod \"13ef0f94-811c-4542-aac5-f4ba52c18f27\" (UID: \"13ef0f94-811c-4542-aac5-f4ba52c18f27\") "
Nov 02 08:15:24.212466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.196923    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ef0f94-811c-4542-aac5-f4ba52c18f27-maas-api-service-ca" pod "13ef0f94-811c-4542-aac5-f4ba52c18f27" (UID: "13ef0f94-811c-4542-aac5-f4ba52c18f27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:24.212466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.211117    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893455-k5mzc"]
Nov 02 08:15:24.217875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.217828    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ef0f94-811c-4542-aac5-f4ba52c18f27-kube-api-access-c9dxc" pod "13ef0f94-811c-4542-aac5-f4ba52c18f27" (UID: "13ef0f94-811c-4542-aac5-f4ba52c18f27"). InnerVolumeSpecName "kube-api-access-c9dxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:24.265514 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.265447    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"]
Nov 02 08:15:24.270776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.270727    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893455-78rbv"]
Nov 02 08:15:24.283590 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.283556    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9dxc\" (UniqueName: \"kubernetes.io/projected/13ef0f94-811c-4542-aac5-f4ba52c18f27-kube-api-access-c9dxc\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:24.283590 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.283587    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ef0f94-811c-4542-aac5-f4ba52c18f27-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:24.353687 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.353656    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"]
Nov 02 08:15:24.355288 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:24.355262    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f504d5_53f9_40ec_bf39_074da613f90b.slice/crio-6d0044955a6dab33d90dd0bceb2f8ec7352117d17869676e5c3527feb4e625c5 WatchSource:0}: Error finding container 6d0044955a6dab33d90dd0bceb2f8ec7352117d17869676e5c3527feb4e625c5: Status 404 returned error can't find the container with id 6d0044955a6dab33d90dd0bceb2f8ec7352117d17869676e5c3527feb4e625c5
Nov 02 08:15:24.809575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.809286    4898 patch_prober.go:29] interesting pod/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:15:24.809575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.809348    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:15:24.871406 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.871363    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6" event={"ID":"74f504d5-53f9-40ec-bf39-074da613f90b","Type":"ContainerStarted","Data":"c8231558612bd56bd63e773c7d350ac13b45c6e9350e56fa1bda3f44d1dc9dd3"}
Nov 02 08:15:24.871599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.871426    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6" event={"ID":"74f504d5-53f9-40ec-bf39-074da613f90b","Type":"ContainerStarted","Data":"6d0044955a6dab33d90dd0bceb2f8ec7352117d17869676e5c3527feb4e625c5"}
Nov 02 08:15:24.873583 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.873547    4898 generic.go:356] "Generic (PLEG): container finished" podID="0d2895ef-71fb-4097-829d-bfb60039769f" containerID="6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116" exitCode=0
Nov 02 08:15:24.873739 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.873609    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" event={"ID":"0d2895ef-71fb-4097-829d-bfb60039769f","Type":"ContainerDied","Data":"6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116"}
Nov 02 08:15:24.874120 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.874101    4898 scope.go:122] "RemoveContainer" containerID="6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116"
Nov 02 08:15:24.877263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.877229    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:15:24.877393 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.877322    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:15:24.877393 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.877377    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:15:24.878292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.878260    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd" event={"ID":"13ef0f94-811c-4542-aac5-f4ba52c18f27","Type":"ContainerDied","Data":"2141a15d0192e84a2eff63bd509d1bc801ca94d5aaa584c68847f24280d11d05"}
Nov 02 08:15:24.878388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.878266    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"
Nov 02 08:15:24.878447 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.878333    4898 scope.go:122] "RemoveContainer" containerID="d3bbaee5192c13cd84100618d7d9ed3443352ad56c06722f09c3d36ee463d26c"
Nov 02 08:15:24.918197 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.918134    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6" podStartSLOduration=1.91811622 podStartE2EDuration="1.91811622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:24.913311902 +0000 UTC m=+3075.603638866" watchObservedRunningTime="2026-11-02 08:15:24.91811622 +0000 UTC m=+3075.608443185"
Nov 02 08:15:24.952486 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.952456    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"]
Nov 02 08:15:24.958878 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:24.958848    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-354660-29893455-77mpd"]
Nov 02 08:15:25.469645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:25.469602    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27" path="/var/lib/kubelet/pods/13ef0f94-811c-4542-aac5-f4ba52c18f27/volumes"
Nov 02 08:15:25.470252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:25.470227    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" path="/var/lib/kubelet/pods/a6cfc1ff-3ea1-4845-98c2-d02d836507a2/volumes"
Nov 02 08:15:25.484824 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:25.484782    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8186095-8ca2-42ab-aedf-d0d44811258a" path="/var/lib/kubelet/pods/d8186095-8ca2-42ab-aedf-d0d44811258a/volumes"
Nov 02 08:15:25.486848 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:25.486817    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95edc7f-4b4b-449d-870c-156648685af2" path="/var/lib/kubelet/pods/d95edc7f-4b4b-449d-870c-156648685af2/volumes"
Nov 02 08:15:25.900271 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:25.900223    4898 generic.go:356] "Generic (PLEG): container finished" podID="74f504d5-53f9-40ec-bf39-074da613f90b" containerID="c8231558612bd56bd63e773c7d350ac13b45c6e9350e56fa1bda3f44d1dc9dd3" exitCode=0
Nov 02 08:15:25.900999 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:25.900954    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6" event={"ID":"74f504d5-53f9-40ec-bf39-074da613f90b","Type":"ContainerDied","Data":"c8231558612bd56bd63e773c7d350ac13b45c6e9350e56fa1bda3f44d1dc9dd3"}
Nov 02 08:15:25.908077 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:25.908034    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" event={"ID":"0d2895ef-71fb-4097-829d-bfb60039769f","Type":"ContainerStarted","Data":"6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c"}
Nov 02 08:15:26.597167 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:26.597129    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:15:26.597461 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:26.597430    4898 patch_prober.go:29] interesting pod/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 02 08:15:26.597550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:26.597507    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 02 08:15:27.146372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.146341    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:27.232434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.232353    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f504d5-53f9-40ec-bf39-074da613f90b-kube-api-access-rg7gn\" (UniqueName: \"kubernetes.io/projected/74f504d5-53f9-40ec-bf39-074da613f90b-kube-api-access-rg7gn\") pod \"74f504d5-53f9-40ec-bf39-074da613f90b\" (UID: \"74f504d5-53f9-40ec-bf39-074da613f90b\") "
Nov 02 08:15:27.232607 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.232477    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-xks-ca\") pod \"74f504d5-53f9-40ec-bf39-074da613f90b\" (UID: \"74f504d5-53f9-40ec-bf39-074da613f90b\") "
Nov 02 08:15:27.232607 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.232595    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-openshift-ca\") pod \"74f504d5-53f9-40ec-bf39-074da613f90b\" (UID: \"74f504d5-53f9-40ec-bf39-074da613f90b\") "
Nov 02 08:15:27.233066 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.233031    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-openshift-ca" pod "74f504d5-53f9-40ec-bf39-074da613f90b" (UID: "74f504d5-53f9-40ec-bf39-074da613f90b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:27.233395 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.233362    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:27.234767 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.234733    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f504d5-53f9-40ec-bf39-074da613f90b-kube-api-access-rg7gn" pod "74f504d5-53f9-40ec-bf39-074da613f90b" (UID: "74f504d5-53f9-40ec-bf39-074da613f90b"). InnerVolumeSpecName "kube-api-access-rg7gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:27.235089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.235071    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-xks-ca" pod "74f504d5-53f9-40ec-bf39-074da613f90b" (UID: "74f504d5-53f9-40ec-bf39-074da613f90b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:15:27.334561 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.334524    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg7gn\" (UniqueName: \"kubernetes.io/projected/74f504d5-53f9-40ec-bf39-074da613f90b-kube-api-access-rg7gn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:27.334561 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.334558    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f504d5-53f9-40ec-bf39-074da613f90b-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:27.600868 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.597870    4898 patch_prober.go:29] interesting pod/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 02 08:15:27.600868 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.597973    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 02 08:15:27.791811 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.791683    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr" podUID="074fba4b-7b02-46cf-9aab-4a57a38968d1" containerName="payload-processing" containerID="cri-o://8da540b60c0d4ab101545d760ee0e4a344976550718d171672b2248f6e2e498e" gracePeriod=25
Nov 02 08:15:27.934160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.934127    4898 generic.go:356] "Generic (PLEG): container finished" podID="074fba4b-7b02-46cf-9aab-4a57a38968d1" containerID="8da540b60c0d4ab101545d760ee0e4a344976550718d171672b2248f6e2e498e" exitCode=0
Nov 02 08:15:27.934421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.934378    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr" event={"ID":"074fba4b-7b02-46cf-9aab-4a57a38968d1","Type":"ContainerDied","Data":"8da540b60c0d4ab101545d760ee0e4a344976550718d171672b2248f6e2e498e"}
Nov 02 08:15:27.935947 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.935880    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6" event={"ID":"74f504d5-53f9-40ec-bf39-074da613f90b","Type":"ContainerDied","Data":"6d0044955a6dab33d90dd0bceb2f8ec7352117d17869676e5c3527feb4e625c5"}
Nov 02 08:15:27.936075 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.935971    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d0044955a6dab33d90dd0bceb2f8ec7352117d17869676e5c3527feb4e625c5"
Nov 02 08:15:27.936123 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:27.936074    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"
Nov 02 08:15:28.152815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.152786    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:28.251088 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.251040    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-plugins-config-volume\") pod \"074fba4b-7b02-46cf-9aab-4a57a38968d1\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") "
Nov 02 08:15:28.251303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.251131    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-kube-api-access-6drx4\" (UniqueName: \"kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-kube-api-access-6drx4\") pod \"074fba4b-7b02-46cf-9aab-4a57a38968d1\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") "
Nov 02 08:15:28.251303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.251162    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-routing-overlay-volume\") pod \"074fba4b-7b02-46cf-9aab-4a57a38968d1\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") "
Nov 02 08:15:28.251445 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.251333    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-provider-credentials\") pod \"074fba4b-7b02-46cf-9aab-4a57a38968d1\" (UID: \"074fba4b-7b02-46cf-9aab-4a57a38968d1\") "
Nov 02 08:15:28.251559 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.251531    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-plugins-config-volume" pod "074fba4b-7b02-46cf-9aab-4a57a38968d1" (UID: "074fba4b-7b02-46cf-9aab-4a57a38968d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:28.252030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.251998    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-routing-overlay-volume" pod "074fba4b-7b02-46cf-9aab-4a57a38968d1" (UID: "074fba4b-7b02-46cf-9aab-4a57a38968d1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:28.252285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.252259    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:28.252285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.252286    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/074fba4b-7b02-46cf-9aab-4a57a38968d1-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:28.254187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.254142    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-provider-credentials" pod "074fba4b-7b02-46cf-9aab-4a57a38968d1" (UID: "074fba4b-7b02-46cf-9aab-4a57a38968d1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:28.254341 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.254308    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-kube-api-access-6drx4" pod "074fba4b-7b02-46cf-9aab-4a57a38968d1" (UID: "074fba4b-7b02-46cf-9aab-4a57a38968d1"). InnerVolumeSpecName "kube-api-access-6drx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:28.289209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.289080    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"
Nov 02 08:15:28.354374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.354330    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6drx4\" (UniqueName: \"kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-kube-api-access-6drx4\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:28.354374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.354377    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/074fba4b-7b02-46cf-9aab-4a57a38968d1-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:28.596767 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.596728    4898 patch_prober.go:29] interesting pod/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 02 08:15:28.597019 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.596795    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 02 08:15:28.833621 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.833550    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6" podUID="6ff85201-d3aa-4dcc-ae86-2a7865b44d82" containerName="payload-processing" containerID="cri-o://82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46" gracePeriod=25
Nov 02 08:15:28.947704 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.947669    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"
Nov 02 08:15:28.947905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.947670    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr" event={"ID":"074fba4b-7b02-46cf-9aab-4a57a38968d1","Type":"ContainerDied","Data":"6ee93631ea3269953730ff1e98d04bcc1fc1c9146c57ecc0b27c015a329fb743"}
Nov 02 08:15:28.947905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:28.947850    4898 scope.go:122] "RemoveContainer" containerID="8da540b60c0d4ab101545d760ee0e4a344976550718d171672b2248f6e2e498e"
Nov 02 08:15:29.025938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.025881    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"]
Nov 02 08:15:29.036927 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.036866    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ea9556/payload-processing-external-model-e2e-extmod-a-ea9556-674czghdr"]
Nov 02 08:15:29.210418 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.210389    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:29.298032 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.297987    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-kube-api-access-9gw7t\" (UniqueName: \"kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-kube-api-access-9gw7t\") pod \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") "
Nov 02 08:15:29.300001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.299345    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-provider-credentials\") pod \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") "
Nov 02 08:15:29.300361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.300334    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-plugins-config-volume\") pod \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") "
Nov 02 08:15:29.300678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.300656    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-routing-overlay-volume\") pod \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\" (UID: \"6ff85201-d3aa-4dcc-ae86-2a7865b44d82\") "
Nov 02 08:15:29.300819 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.300791    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-plugins-config-volume" pod "6ff85201-d3aa-4dcc-ae86-2a7865b44d82" (UID: "6ff85201-d3aa-4dcc-ae86-2a7865b44d82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:29.301291 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.301263    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-routing-overlay-volume" pod "6ff85201-d3aa-4dcc-ae86-2a7865b44d82" (UID: "6ff85201-d3aa-4dcc-ae86-2a7865b44d82"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:29.301774 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.301748    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:29.301988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.301972    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:29.303202 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.303171    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-kube-api-access-9gw7t" pod "6ff85201-d3aa-4dcc-ae86-2a7865b44d82" (UID: "6ff85201-d3aa-4dcc-ae86-2a7865b44d82"). InnerVolumeSpecName "kube-api-access-9gw7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:29.303499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.303465    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-provider-credentials" pod "6ff85201-d3aa-4dcc-ae86-2a7865b44d82" (UID: "6ff85201-d3aa-4dcc-ae86-2a7865b44d82"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:29.423997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.414819    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gw7t\" (UniqueName: \"kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-kube-api-access-9gw7t\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:29.423997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.414873    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ff85201-d3aa-4dcc-ae86-2a7865b44d82-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:29.494831 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.494784    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="074fba4b-7b02-46cf-9aab-4a57a38968d1" path="/var/lib/kubelet/pods/074fba4b-7b02-46cf-9aab-4a57a38968d1/volumes"
Nov 02 08:15:29.596091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.596052    4898 patch_prober.go:29] interesting pod/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 02 08:15:29.596296 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.596112    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 02 08:15:29.983489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.983438    4898 generic.go:356] "Generic (PLEG): container finished" podID="6ff85201-d3aa-4dcc-ae86-2a7865b44d82" containerID="82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46" exitCode=0
Nov 02 08:15:29.983489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.983482    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"
Nov 02 08:15:29.983781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.983521    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6" event={"ID":"6ff85201-d3aa-4dcc-ae86-2a7865b44d82","Type":"ContainerDied","Data":"82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46"}
Nov 02 08:15:29.983781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.983578    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6" event={"ID":"6ff85201-d3aa-4dcc-ae86-2a7865b44d82","Type":"ContainerDied","Data":"bc3e8bb4321913b603978592cda3116eb6a6867037f2c750f5e44d1961a27f19"}
Nov 02 08:15:29.983781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.983608    4898 scope.go:122] "RemoveContainer" containerID="82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46"
Nov 02 08:15:29.995615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.995586    4898 scope.go:122] "RemoveContainer" containerID="82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46"
Nov 02 08:15:29.995921 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:29.995898    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46\": container with ID starting with 82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46 not found: ID does not exist" containerID="82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46"
Nov 02 08:15:29.995974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:29.995932    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46"} err="failed to get container status \"82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46\": rpc error: code = NotFound desc = could not find container \"82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46\": container with ID starting with 82838efacda751eef116047bc90e6cd11c2b409797b36cd8237e40a10b8ead46 not found: ID does not exist"
Nov 02 08:15:30.038858 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:30.038817    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"]
Nov 02 08:15:30.051147 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:30.051117    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-5fdcvshk6"]
Nov 02 08:15:30.596511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:30.596475    4898 patch_prober.go:29] interesting pod/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 02 08:15:30.597114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:30.596546    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 02 08:15:31.471548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:31.471497    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff85201-d3aa-4dcc-ae86-2a7865b44d82" path="/var/lib/kubelet/pods/6ff85201-d3aa-4dcc-ae86-2a7865b44d82/volumes"
Nov 02 08:15:31.600732 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:31.600695    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:15:31.601259 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:31.601243    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:15:31.602086 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:31.602067    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:15:33.818786 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.818747    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"]
Nov 02 08:15:33.819541 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819526    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ee823ad5-4625-4083-9cc9-20044043dedf" containerName="cleanup"
Nov 02 08:15:33.819591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819543    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee823ad5-4625-4083-9cc9-20044043dedf" containerName="cleanup"
Nov 02 08:15:33.819591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819561    4898 cpu_manager.go:414] "containerMap: removing container" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27" containerName="cleanup"
Nov 02 08:15:33.819591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819567    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27" containerName="cleanup"
Nov 02 08:15:33.819591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819582    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41" containerName="cleanup"
Nov 02 08:15:33.819591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819587    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819596    4898 cpu_manager.go:414] "containerMap: removing container" podUID="074fba4b-7b02-46cf-9aab-4a57a38968d1" containerName="payload-processing"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819601    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074fba4b-7b02-46cf-9aab-4a57a38968d1" containerName="payload-processing"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819619    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819623    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819634    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819639    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819645    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819650    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819656    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819660    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819670    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819674    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819681    4898 cpu_manager.go:414] "containerMap: removing container" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819685    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819693    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d95edc7f-4b4b-449d-870c-156648685af2" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819697    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95edc7f-4b4b-449d-870c-156648685af2" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819706    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819711    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819719    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d95edc7f-4b4b-449d-870c-156648685af2" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819724    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95edc7f-4b4b-449d-870c-156648685af2" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819732    4898 cpu_manager.go:414] "containerMap: removing container" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819737    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819745    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1f648095-5449-42df-885a-29e8044cb8c2" containerName="cleanup"
Nov 02 08:15:33.819749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819749    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f648095-5449-42df-885a-29e8044cb8c2" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819762    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6ff85201-d3aa-4dcc-ae86-2a7865b44d82" containerName="payload-processing"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819768    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff85201-d3aa-4dcc-ae86-2a7865b44d82" containerName="payload-processing"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819778    4898 cpu_manager.go:414] "containerMap: removing container" podUID="74f504d5-53f9-40ec-bf39-074da613f90b" containerName="revoke-keys"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819785    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f504d5-53f9-40ec-bf39-074da613f90b" containerName="revoke-keys"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819910    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee823ad5-4625-4083-9cc9-20044043dedf" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819923    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819930    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0615ef0f-a1d8-47cc-a6b4-481fadc306bc" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819936    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2876d3b-6372-4dd1-919d-4a53af9540bb" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819945    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2da54f0-7885-4cd9-969c-c6051ba28f41" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819951    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81cf60f-13a6-4260-9364-114e89edeef7" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819956    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="074fba4b-7b02-46cf-9aab-4a57a38968d1" containerName="payload-processing"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819964    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bca6afb-be58-48e6-8d4f-317c45edcffd" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819972    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819977    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819984    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab5290b-4dd7-494c-9381-a467e78f96c8" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819992    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95edc7f-4b4b-449d-870c-156648685af2" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.819996    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9584c1c3-65cf-4e75-aa0f-9a3681010cc2" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.820005    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff85201-d3aa-4dcc-ae86-2a7865b44d82" containerName="payload-processing"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.820011    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95edc7f-4b4b-449d-870c-156648685af2" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.820016    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.820021    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f504d5-53f9-40ec-bf39-074da613f90b" containerName="revoke-keys"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.820027    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f648095-5449-42df-885a-29e8044cb8c2" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.820033    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff46f33-1e5a-4a41-85a1-7d18107f3844" containerName="cleanup"
Nov 02 08:15:33.820562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.820038    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf3abb8-f4bd-4770-8ce7-5df5defd7fb5" containerName="cleanup"
Nov 02 08:15:33.825754 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.825734    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:33.845292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.845260    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"]
Nov 02 08:15:33.847668 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.847636    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:15:33.847795 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.847691    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:15:33.847918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.847866    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:15:33.847974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.847958    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:15:33.893305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.893268    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6bm\" (UniqueName: \"kubernetes.io/projected/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-kube-api-access-nb6bm\") pod \"maas-api-revoke-sub-22c93a6d-8bn9c\" (UID: \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:33.893515 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.893423    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c93a6d-8bn9c\" (UID: \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:33.893651 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.893629    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c93a6d-8bn9c\" (UID: \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:33.995785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.995733    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6bm\" (UniqueName: \"kubernetes.io/projected/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-kube-api-access-nb6bm\") pod \"maas-api-revoke-sub-22c93a6d-8bn9c\" (UID: \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:33.995985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.995866    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c93a6d-8bn9c\" (UID: \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:33.996097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.996077    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c93a6d-8bn9c\" (UID: \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:33.996718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.996692    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c93a6d-8bn9c\" (UID: \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:33.998436 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:33.998412    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c93a6d-8bn9c\" (UID: \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:34.019981 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:34.019940    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6bm\" (UniqueName: \"kubernetes.io/projected/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-kube-api-access-nb6bm\") pod \"maas-api-revoke-sub-22c93a6d-8bn9c\" (UID: \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:34.145964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:34.145866    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:34.455613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:34.455581    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"]
Nov 02 08:15:34.456304 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:34.456269    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e20572_c6b6_4a4f_9519_873d9b7a2b7a.slice/crio-98b3d09d2a077678a33fd72beb4da5e17768b328c170441ef4c240b6e8137170 WatchSource:0}: Error finding container 98b3d09d2a077678a33fd72beb4da5e17768b328c170441ef4c240b6e8137170: Status 404 returned error can't find the container with id 98b3d09d2a077678a33fd72beb4da5e17768b328c170441ef4c240b6e8137170
Nov 02 08:15:35.045980 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:35.045931    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c" event={"ID":"17e20572-c6b6-4a4f-9519-873d9b7a2b7a","Type":"ContainerStarted","Data":"0305ed181b7ec9aeef00603f005368787f26269bb552f00d69f74eebcaabab92"}
Nov 02 08:15:35.046435 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:35.046034    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c" event={"ID":"17e20572-c6b6-4a4f-9519-873d9b7a2b7a","Type":"ContainerStarted","Data":"98b3d09d2a077678a33fd72beb4da5e17768b328c170441ef4c240b6e8137170"}
Nov 02 08:15:35.091491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:35.091429    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c" podStartSLOduration=2.091412369 podStartE2EDuration="2.091412369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:35.086943008 +0000 UTC m=+3085.777269979" watchObservedRunningTime="2026-11-02 08:15:35.091412369 +0000 UTC m=+3085.781739333"
Nov 02 08:15:36.058597 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:36.058558    4898 generic.go:356] "Generic (PLEG): container finished" podID="17e20572-c6b6-4a4f-9519-873d9b7a2b7a" containerID="0305ed181b7ec9aeef00603f005368787f26269bb552f00d69f74eebcaabab92" exitCode=0
Nov 02 08:15:36.059048 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:36.058642    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c" event={"ID":"17e20572-c6b6-4a4f-9519-873d9b7a2b7a","Type":"ContainerDied","Data":"0305ed181b7ec9aeef00603f005368787f26269bb552f00d69f74eebcaabab92"}
Nov 02 08:15:37.266809 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:37.266779    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:37.355582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:37.355471    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-xks-ca\") pod \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\" (UID: \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\") "
Nov 02 08:15:37.355773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:37.355660    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-openshift-ca\") pod \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\" (UID: \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\") "
Nov 02 08:15:37.355885 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:37.355868    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-kube-api-access-nb6bm\" (UniqueName: \"kubernetes.io/projected/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-kube-api-access-nb6bm\") pod \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\" (UID: \"17e20572-c6b6-4a4f-9519-873d9b7a2b7a\") "
Nov 02 08:15:37.356385 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:37.356347    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-openshift-ca" pod "17e20572-c6b6-4a4f-9519-873d9b7a2b7a" (UID: "17e20572-c6b6-4a4f-9519-873d9b7a2b7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:37.356816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:37.356787    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:37.358754 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:37.358723    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-xks-ca" pod "17e20572-c6b6-4a4f-9519-873d9b7a2b7a" (UID: "17e20572-c6b6-4a4f-9519-873d9b7a2b7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:15:37.358924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:37.358779    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-kube-api-access-nb6bm" pod "17e20572-c6b6-4a4f-9519-873d9b7a2b7a" (UID: "17e20572-c6b6-4a4f-9519-873d9b7a2b7a"). InnerVolumeSpecName "kube-api-access-nb6bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:37.458290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:37.458249    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb6bm\" (UniqueName: \"kubernetes.io/projected/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-kube-api-access-nb6bm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:37.458290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:37.458289    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e20572-c6b6-4a4f-9519-873d9b7a2b7a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:38.078975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:38.078940    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"
Nov 02 08:15:38.078975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:38.078958    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c" event={"ID":"17e20572-c6b6-4a4f-9519-873d9b7a2b7a","Type":"ContainerDied","Data":"98b3d09d2a077678a33fd72beb4da5e17768b328c170441ef4c240b6e8137170"}
Nov 02 08:15:38.079275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:38.079021    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98b3d09d2a077678a33fd72beb4da5e17768b328c170441ef4c240b6e8137170"
Nov 02 08:15:44.993283 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:44.993232    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:15:47.185958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:47.183147    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"]
Nov 02 08:15:48.186437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.186397    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"]
Nov 02 08:15:48.187697 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.187668    4898 cpu_manager.go:414] "containerMap: removing container" podUID="17e20572-c6b6-4a4f-9519-873d9b7a2b7a" containerName="revoke-keys"
Nov 02 08:15:48.187697 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.187695    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e20572-c6b6-4a4f-9519-873d9b7a2b7a" containerName="revoke-keys"
Nov 02 08:15:48.187955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.187712    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" containerName="cleanup"
Nov 02 08:15:48.187955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.187720    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" containerName="cleanup"
Nov 02 08:15:48.187955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.187738    4898 cpu_manager.go:414] "containerMap: removing container" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27" containerName="cleanup"
Nov 02 08:15:48.187955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.187747    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27" containerName="cleanup"
Nov 02 08:15:48.187955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.187766    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d95edc7f-4b4b-449d-870c-156648685af2" containerName="cleanup"
Nov 02 08:15:48.187955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.187773    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95edc7f-4b4b-449d-870c-156648685af2" containerName="cleanup"
Nov 02 08:15:48.188304 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.187978    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ef0f94-811c-4542-aac5-f4ba52c18f27" containerName="cleanup"
Nov 02 08:15:48.188304 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.187999    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e20572-c6b6-4a4f-9519-873d9b7a2b7a" containerName="revoke-keys"
Nov 02 08:15:48.188304 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.188009    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6cfc1ff-3ea1-4845-98c2-d02d836507a2" containerName="cleanup"
Nov 02 08:15:48.188304 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.188022    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95edc7f-4b4b-449d-870c-156648685af2" containerName="cleanup"
Nov 02 08:15:48.193273 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.193249    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.213378 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.213345    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"]
Nov 02 08:15:48.213815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.213791    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dfd6f0-openshift-default-dockercfg-2b8t8\""
Nov 02 08:15:48.375913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.335879    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-envoy\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.375913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.336029    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-credential-socket\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.375913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.336103    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-data\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.375913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.336237    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9r7v\" (UniqueName: \"kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-kube-api-access-l9r7v\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.375913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.336397    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1900eb00-365d-401d-a399-946fb0c03bfa-istiod-ca-cert\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.375913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.336425    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-istio-token\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.375913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.336516    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-certs\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.375913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.336556    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1900eb00-365d-401d-a399-946fb0c03bfa-istio-podinfo\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.375913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.336623    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-socket\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.437946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.437829    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9r7v\" (UniqueName: \"kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-kube-api-access-l9r7v\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.452030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.451989    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1900eb00-365d-401d-a399-946fb0c03bfa-istiod-ca-cert\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.453191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.453167    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-istio-token\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.453767 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.453743    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-certs\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.454257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.454240    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1900eb00-365d-401d-a399-946fb0c03bfa-istio-podinfo\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.468395 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.468361    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-socket\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.468945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.453099    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1900eb00-365d-401d-a399-946fb0c03bfa-istiod-ca-cert\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.469548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.468868    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-socket\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.469714 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.454186    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-certs\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.469847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.469157    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-envoy\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.470042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.470024    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-credential-socket\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.470508 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.470397    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-credential-socket\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.471003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.470963    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-data\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.471122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.470667    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-data\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.499257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.499193    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-envoy\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.500300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.500015    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1900eb00-365d-401d-a399-946fb0c03bfa-istio-podinfo\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.512909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.500655    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9r7v\" (UniqueName: \"kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-kube-api-access-l9r7v\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.513128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.507589    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-istio-token\") pod \"e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") " pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.524561 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.523169    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:48.995389 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:48.986170    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"]
Nov 02 08:15:49.010707 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:49.010641    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1900eb00_365d_401d_a399_946fb0c03bfa.slice/crio-3f2a70fc1284aa29da0cab511f0aeedb8d9eddc334d911ddc201e4717c571547 WatchSource:0}: Error finding container 3f2a70fc1284aa29da0cab511f0aeedb8d9eddc334d911ddc201e4717c571547: Status 404 returned error can't find the container with id 3f2a70fc1284aa29da0cab511f0aeedb8d9eddc334d911ddc201e4717c571547
Nov 02 08:15:49.016658 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:49.016606    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:15:49.016778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:49.016715    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:15:49.016778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:49.016757    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:15:49.230351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:49.230307    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" event={"ID":"1900eb00-365d-401d-a399-946fb0c03bfa","Type":"ContainerStarted","Data":"f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c"}
Nov 02 08:15:49.230771 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:49.230389    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" event={"ID":"1900eb00-365d-401d-a399-946fb0c03bfa","Type":"ContainerStarted","Data":"3f2a70fc1284aa29da0cab511f0aeedb8d9eddc334d911ddc201e4717c571547"}
Nov 02 08:15:49.279746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:49.279531    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" podStartSLOduration=1.279513426 podStartE2EDuration="1.279513426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:49.275068326 +0000 UTC m=+3099.965395313" watchObservedRunningTime="2026-11-02 08:15:49.279513426 +0000 UTC m=+3099.969840574"
Nov 02 08:15:49.524519 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:49.524488    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:50.767597 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:50.767564    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:51.253691 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:51.253655    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:51.254997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:51.254968    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:15:52.184968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.183976    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt" podUID="ab522265-0276-413e-99f0-c8266c417bd5" containerName="payload-processing" containerID="cri-o://aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44" gracePeriod=25
Nov 02 08:15:52.619442 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.619399    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:15:52.651551 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.651514    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-routing-overlay-volume\") pod \"ab522265-0276-413e-99f0-c8266c417bd5\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") "
Nov 02 08:15:52.651756 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.651677    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-plugins-config-volume\") pod \"ab522265-0276-413e-99f0-c8266c417bd5\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") "
Nov 02 08:15:52.651857 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.651841    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-kube-api-access-kbdpp\" (UniqueName: \"kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-kube-api-access-kbdpp\") pod \"ab522265-0276-413e-99f0-c8266c417bd5\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") "
Nov 02 08:15:52.651995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.651966    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-routing-overlay-volume" pod "ab522265-0276-413e-99f0-c8266c417bd5" (UID: "ab522265-0276-413e-99f0-c8266c417bd5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:52.652329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.652304    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-plugins-config-volume" pod "ab522265-0276-413e-99f0-c8266c417bd5" (UID: "ab522265-0276-413e-99f0-c8266c417bd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:52.652469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.652378    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-provider-credentials\") pod \"ab522265-0276-413e-99f0-c8266c417bd5\" (UID: \"ab522265-0276-413e-99f0-c8266c417bd5\") "
Nov 02 08:15:52.653192 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.653126    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:52.653192 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.653152    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab522265-0276-413e-99f0-c8266c417bd5-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:52.654675 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.654650    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-kube-api-access-kbdpp" pod "ab522265-0276-413e-99f0-c8266c417bd5" (UID: "ab522265-0276-413e-99f0-c8266c417bd5"). InnerVolumeSpecName "kube-api-access-kbdpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:52.654820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.654749    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-provider-credentials" pod "ab522265-0276-413e-99f0-c8266c417bd5" (UID: "ab522265-0276-413e-99f0-c8266c417bd5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:52.754799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.754710    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbdpp\" (UniqueName: \"kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-kube-api-access-kbdpp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:52.754799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:52.754747    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab522265-0276-413e-99f0-c8266c417bd5-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:53.380960 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.380882    4898 generic.go:356] "Generic (PLEG): container finished" podID="ab522265-0276-413e-99f0-c8266c417bd5" containerID="aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44" exitCode=0
Nov 02 08:15:53.381419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.380959    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"
Nov 02 08:15:53.381419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.380980    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt" event={"ID":"ab522265-0276-413e-99f0-c8266c417bd5","Type":"ContainerDied","Data":"aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44"}
Nov 02 08:15:53.381419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.381039    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt" event={"ID":"ab522265-0276-413e-99f0-c8266c417bd5","Type":"ContainerDied","Data":"8c4688e68ae5a2bac9cb084c3f87c9f87c2f44e278c86564b951024fa3bebe1d"}
Nov 02 08:15:53.381419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.381068    4898 scope.go:122] "RemoveContainer" containerID="aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44"
Nov 02 08:15:53.394664 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.394638    4898 scope.go:122] "RemoveContainer" containerID="aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44"
Nov 02 08:15:53.394996 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:15:53.394974    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44\": container with ID starting with aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44 not found: ID does not exist" containerID="aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44"
Nov 02 08:15:53.395060 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.395005    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44"} err="failed to get container status \"aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44\": rpc error: code = NotFound desc = could not find container \"aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44\": container with ID starting with aa20abcf220a9f15dcfef519699731cbc4e18492ce6d38f20de80ac927ed7b44 not found: ID does not exist"
Nov 02 08:15:53.439101 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.439065    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"]
Nov 02 08:15:53.451420 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.451384    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-71a41c/payload-processing-external-model-e2e-extmod-b-71a41c-7fd4vkdqt"]
Nov 02 08:15:53.468979 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.468869    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab522265-0276-413e-99f0-c8266c417bd5" path="/var/lib/kubelet/pods/ab522265-0276-413e-99f0-c8266c417bd5/volumes"
Nov 02 08:15:53.696814 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.696769    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"]
Nov 02 08:15:53.698184 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.698152    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ab522265-0276-413e-99f0-c8266c417bd5" containerName="payload-processing"
Nov 02 08:15:53.698184 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.698181    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab522265-0276-413e-99f0-c8266c417bd5" containerName="payload-processing"
Nov 02 08:15:53.698417 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.698349    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab522265-0276-413e-99f0-c8266c417bd5" containerName="payload-processing"
Nov 02 08:15:53.707200 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.707171    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.730177 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.729285    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e6f4fe-openshift-default-dockercfg-z9wmn\""
Nov 02 08:15:53.744954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.741383    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"]
Nov 02 08:15:53.781939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.781899    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istiod-ca-cert\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.782153 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.782096    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-certs\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.782153 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.782129    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-podinfo\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.782302 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.782172    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-data\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.782372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.782295    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-envoy\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.782478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.782456    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6v6k\" (UniqueName: \"kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-kube-api-access-d6v6k\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.782549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.782497    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-credential-socket\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.782610 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.782575    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-socket\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.782610 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.782596    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-token\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.884524 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.884487    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-certs\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.884524 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.884527    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-podinfo\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.884759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.884645    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-data\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.884759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.884687    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-envoy\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.884905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.884796    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6v6k\" (UniqueName: \"kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-kube-api-access-d6v6k\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.884905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.884830    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-credential-socket\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.884905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.884866    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-socket\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.885093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.884937    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-token\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.885093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.884987    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-certs\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.885093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.885059    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istiod-ca-cert\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.885093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.885066    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-data\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.885325 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.885250    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-socket\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.885456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.885430    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-credential-socket\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.885852 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.885829    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istiod-ca-cert\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.887306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.887275    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-envoy\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.887644 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.887619    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-podinfo\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.911656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.911623    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-token\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:53.915762 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:53.915732    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6v6k\" (UniqueName: \"kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-kube-api-access-d6v6k\") pod \"e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") " pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:54.074963 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:54.063351    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:54.597000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:54.596966    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"]
Nov 02 08:15:54.598312 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:54.598281    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded94547d_f04f_4401_a356_fd38ffdcf2e4.slice/crio-6d8445317b9af1c5b65af816fe65ef1ed65ae70ebbfe8d8ad9ff6fdc5a56a6eb WatchSource:0}: Error finding container 6d8445317b9af1c5b65af816fe65ef1ed65ae70ebbfe8d8ad9ff6fdc5a56a6eb: Status 404 returned error can't find the container with id 6d8445317b9af1c5b65af816fe65ef1ed65ae70ebbfe8d8ad9ff6fdc5a56a6eb
Nov 02 08:15:54.601976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:54.601930    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:15:54.602078 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:54.602032    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:15:54.602131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:54.602076    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:15:54.896681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:54.896586    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"]
Nov 02 08:15:54.902697 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:54.901759    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:54.917036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:54.916913    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:15:54.917036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:54.916966    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:15:54.917036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:54.917012    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:15:54.917318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:54.917118    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:15:54.923003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:54.922972    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"]
Nov 02 08:15:55.012945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.012880    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0703208e-mdd8b\" (UID: \"513e3508-7224-4fc1-ad0c-13db184d0934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:55.013129 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.012967    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9szhj\" (UniqueName: \"kubernetes.io/projected/513e3508-7224-4fc1-ad0c-13db184d0934-kube-api-access-9szhj\") pod \"maas-api-revoke-sub-0703208e-mdd8b\" (UID: \"513e3508-7224-4fc1-ad0c-13db184d0934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:55.013129 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.013115    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0703208e-mdd8b\" (UID: \"513e3508-7224-4fc1-ad0c-13db184d0934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:55.123912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.121197    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0703208e-mdd8b\" (UID: \"513e3508-7224-4fc1-ad0c-13db184d0934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:55.123912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.121277    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9szhj\" (UniqueName: \"kubernetes.io/projected/513e3508-7224-4fc1-ad0c-13db184d0934-kube-api-access-9szhj\") pod \"maas-api-revoke-sub-0703208e-mdd8b\" (UID: \"513e3508-7224-4fc1-ad0c-13db184d0934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:55.123912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.121537    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0703208e-mdd8b\" (UID: \"513e3508-7224-4fc1-ad0c-13db184d0934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:55.137942 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.135829    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0703208e-mdd8b\" (UID: \"513e3508-7224-4fc1-ad0c-13db184d0934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:55.152978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.150599    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0703208e-mdd8b\" (UID: \"513e3508-7224-4fc1-ad0c-13db184d0934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:55.173193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.171629    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9szhj\" (UniqueName: \"kubernetes.io/projected/513e3508-7224-4fc1-ad0c-13db184d0934-kube-api-access-9szhj\") pod \"maas-api-revoke-sub-0703208e-mdd8b\" (UID: \"513e3508-7224-4fc1-ad0c-13db184d0934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:55.226971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.226928    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:55.414745 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.414650    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" event={"ID":"ed94547d-f04f-4401-a356-fd38ffdcf2e4","Type":"ContainerStarted","Data":"4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37"}
Nov 02 08:15:55.414745 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.414715    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" event={"ID":"ed94547d-f04f-4401-a356-fd38ffdcf2e4","Type":"ContainerStarted","Data":"6d8445317b9af1c5b65af816fe65ef1ed65ae70ebbfe8d8ad9ff6fdc5a56a6eb"}
Nov 02 08:15:55.534364 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.495163    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" podStartSLOduration=2.495139186 podStartE2EDuration="2.495139186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:55.482328694 +0000 UTC m=+3106.172655671" watchObservedRunningTime="2026-11-02 08:15:55.495139186 +0000 UTC m=+3106.185466153"
Nov 02 08:15:55.579646 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:55.578652    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"]
Nov 02 08:15:55.579646 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:55.578776    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513e3508_7224_4fc1_ad0c_13db184d0934.slice/crio-e2b76a2ab518b101e849b130ec86439437bb2d0f5e38c477f8a69d6a4737dea9 WatchSource:0}: Error finding container e2b76a2ab518b101e849b130ec86439437bb2d0f5e38c477f8a69d6a4737dea9: Status 404 returned error can't find the container with id e2b76a2ab518b101e849b130ec86439437bb2d0f5e38c477f8a69d6a4737dea9
Nov 02 08:15:56.068466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:56.068427    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:56.075333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:56.075308    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:56.425827 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:56.425780    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b" event={"ID":"513e3508-7224-4fc1-ad0c-13db184d0934","Type":"ContainerStarted","Data":"e142593ce02195c5d85af2e58799f9f275ce23474fa02d83a3052429cef06bb1"}
Nov 02 08:15:56.426091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:56.425855    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b" event={"ID":"513e3508-7224-4fc1-ad0c-13db184d0934","Type":"ContainerStarted","Data":"e2b76a2ab518b101e849b130ec86439437bb2d0f5e38c477f8a69d6a4737dea9"}
Nov 02 08:15:56.426302 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:56.426280    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:56.427311 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:56.427286    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:15:56.469442 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:56.469380    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b" podStartSLOduration=2.469362425 podStartE2EDuration="2.469362425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:15:56.465689962 +0000 UTC m=+3107.156016930" watchObservedRunningTime="2026-11-02 08:15:56.469362425 +0000 UTC m=+3107.159689390"
Nov 02 08:15:57.438983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.438943    4898 generic.go:356] "Generic (PLEG): container finished" podID="513e3508-7224-4fc1-ad0c-13db184d0934" containerID="e142593ce02195c5d85af2e58799f9f275ce23474fa02d83a3052429cef06bb1" exitCode=0
Nov 02 08:15:57.439490 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.438988    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b" event={"ID":"513e3508-7224-4fc1-ad0c-13db184d0934","Type":"ContainerDied","Data":"e142593ce02195c5d85af2e58799f9f275ce23474fa02d83a3052429cef06bb1"}
Nov 02 08:15:57.642030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.641979    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"]
Nov 02 08:15:57.647140 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.647106    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:15:57.665906 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.665858    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"]
Nov 02 08:15:57.716446 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.716349    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2510c0c-8k6lj\" (UID: \"8fa104e2-5087-434a-ac18-459608d49e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:15:57.716719 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.716697    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2510c0c-8k6lj\" (UID: \"8fa104e2-5087-434a-ac18-459608d49e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:15:57.716844 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.716821    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jqpv\" (UniqueName: \"kubernetes.io/projected/8fa104e2-5087-434a-ac18-459608d49e6f-kube-api-access-7jqpv\") pod \"maas-api-revoke-sub-e2510c0c-8k6lj\" (UID: \"8fa104e2-5087-434a-ac18-459608d49e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:15:57.818867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.818827    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2510c0c-8k6lj\" (UID: \"8fa104e2-5087-434a-ac18-459608d49e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:15:57.819107 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.818970    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jqpv\" (UniqueName: \"kubernetes.io/projected/8fa104e2-5087-434a-ac18-459608d49e6f-kube-api-access-7jqpv\") pod \"maas-api-revoke-sub-e2510c0c-8k6lj\" (UID: \"8fa104e2-5087-434a-ac18-459608d49e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:15:57.819107 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.819086    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2510c0c-8k6lj\" (UID: \"8fa104e2-5087-434a-ac18-459608d49e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:15:57.819601 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.819570    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2510c0c-8k6lj\" (UID: \"8fa104e2-5087-434a-ac18-459608d49e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:15:57.821705 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.821679    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2510c0c-8k6lj\" (UID: \"8fa104e2-5087-434a-ac18-459608d49e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:15:57.845441 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.845403    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jqpv\" (UniqueName: \"kubernetes.io/projected/8fa104e2-5087-434a-ac18-459608d49e6f-kube-api-access-7jqpv\") pod \"maas-api-revoke-sub-e2510c0c-8k6lj\" (UID: \"8fa104e2-5087-434a-ac18-459608d49e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:15:57.972306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:57.972204    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:15:58.299334 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:15:58.299299    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa104e2_5087_434a_ac18_459608d49e6f.slice/crio-eb409633dd8144b61db8bfdaac73567b761dcaa9a06adb9e6679a6fc24aa9a9c WatchSource:0}: Error finding container eb409633dd8144b61db8bfdaac73567b761dcaa9a06adb9e6679a6fc24aa9a9c: Status 404 returned error can't find the container with id eb409633dd8144b61db8bfdaac73567b761dcaa9a06adb9e6679a6fc24aa9a9c
Nov 02 08:15:58.302562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:58.302532    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"]
Nov 02 08:15:58.450959 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:58.450911    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj" event={"ID":"8fa104e2-5087-434a-ac18-459608d49e6f","Type":"ContainerStarted","Data":"eb409633dd8144b61db8bfdaac73567b761dcaa9a06adb9e6679a6fc24aa9a9c"}
Nov 02 08:15:58.624601 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:58.624569    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:58.736256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:58.736221    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-xks-ca\") pod \"513e3508-7224-4fc1-ad0c-13db184d0934\" (UID: \"513e3508-7224-4fc1-ad0c-13db184d0934\") "
Nov 02 08:15:58.736449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:58.736382    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-openshift-ca\") pod \"513e3508-7224-4fc1-ad0c-13db184d0934\" (UID: \"513e3508-7224-4fc1-ad0c-13db184d0934\") "
Nov 02 08:15:58.736449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:58.736431    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/513e3508-7224-4fc1-ad0c-13db184d0934-kube-api-access-9szhj\" (UniqueName: \"kubernetes.io/projected/513e3508-7224-4fc1-ad0c-13db184d0934-kube-api-access-9szhj\") pod \"513e3508-7224-4fc1-ad0c-13db184d0934\" (UID: \"513e3508-7224-4fc1-ad0c-13db184d0934\") "
Nov 02 08:15:58.736770 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:58.736739    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-openshift-ca" pod "513e3508-7224-4fc1-ad0c-13db184d0934" (UID: "513e3508-7224-4fc1-ad0c-13db184d0934"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:15:58.737179 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:58.737160    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:58.738547 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:58.738522    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-xks-ca" pod "513e3508-7224-4fc1-ad0c-13db184d0934" (UID: "513e3508-7224-4fc1-ad0c-13db184d0934"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:15:58.738647 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:58.738616    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513e3508-7224-4fc1-ad0c-13db184d0934-kube-api-access-9szhj" pod "513e3508-7224-4fc1-ad0c-13db184d0934" (UID: "513e3508-7224-4fc1-ad0c-13db184d0934"). InnerVolumeSpecName "kube-api-access-9szhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:15:58.838575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:58.838540    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9szhj\" (UniqueName: \"kubernetes.io/projected/513e3508-7224-4fc1-ad0c-13db184d0934-kube-api-access-9szhj\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:58.838727 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:58.838578    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513e3508-7224-4fc1-ad0c-13db184d0934-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:15:59.465219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:59.465176    4898 generic.go:356] "Generic (PLEG): container finished" podID="8fa104e2-5087-434a-ac18-459608d49e6f" containerID="4c6fea93710bf9715bf423016daae67ff2aa55cf76f167b958dbb97875991f77" exitCode=0
Nov 02 08:15:59.467171 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:59.467147    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"
Nov 02 08:15:59.471013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:59.470982    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj" event={"ID":"8fa104e2-5087-434a-ac18-459608d49e6f","Type":"ContainerDied","Data":"4c6fea93710bf9715bf423016daae67ff2aa55cf76f167b958dbb97875991f77"}
Nov 02 08:15:59.471165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:59.471038    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b" event={"ID":"513e3508-7224-4fc1-ad0c-13db184d0934","Type":"ContainerDied","Data":"e2b76a2ab518b101e849b130ec86439437bb2d0f5e38c477f8a69d6a4737dea9"}
Nov 02 08:15:59.471165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:15:59.471093    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2b76a2ab518b101e849b130ec86439437bb2d0f5e38c477f8a69d6a4737dea9"
Nov 02 08:16:00.687531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:00.687502    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:16:00.724317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:00.724283    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa104e2-5087-434a-ac18-459608d49e6f-kube-api-access-7jqpv\" (UniqueName: \"kubernetes.io/projected/8fa104e2-5087-434a-ac18-459608d49e6f-kube-api-access-7jqpv\") pod \"8fa104e2-5087-434a-ac18-459608d49e6f\" (UID: \"8fa104e2-5087-434a-ac18-459608d49e6f\") "
Nov 02 08:16:00.724528 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:00.724514    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-xks-ca\") pod \"8fa104e2-5087-434a-ac18-459608d49e6f\" (UID: \"8fa104e2-5087-434a-ac18-459608d49e6f\") "
Nov 02 08:16:00.724593 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:00.724574    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-openshift-ca\") pod \"8fa104e2-5087-434a-ac18-459608d49e6f\" (UID: \"8fa104e2-5087-434a-ac18-459608d49e6f\") "
Nov 02 08:16:00.724982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:00.724955    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-openshift-ca" pod "8fa104e2-5087-434a-ac18-459608d49e6f" (UID: "8fa104e2-5087-434a-ac18-459608d49e6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:00.725175 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:00.725142    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:00.727252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:00.727224    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-xks-ca" pod "8fa104e2-5087-434a-ac18-459608d49e6f" (UID: "8fa104e2-5087-434a-ac18-459608d49e6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:16:00.727415 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:00.727391    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa104e2-5087-434a-ac18-459608d49e6f-kube-api-access-7jqpv" pod "8fa104e2-5087-434a-ac18-459608d49e6f" (UID: "8fa104e2-5087-434a-ac18-459608d49e6f"). InnerVolumeSpecName "kube-api-access-7jqpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:00.826328 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:00.826292    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fa104e2-5087-434a-ac18-459608d49e6f-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:00.826328 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:00.826322    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jqpv\" (UniqueName: \"kubernetes.io/projected/8fa104e2-5087-434a-ac18-459608d49e6f-kube-api-access-7jqpv\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:01.522877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:01.522834    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj" event={"ID":"8fa104e2-5087-434a-ac18-459608d49e6f","Type":"ContainerDied","Data":"eb409633dd8144b61db8bfdaac73567b761dcaa9a06adb9e6679a6fc24aa9a9c"}
Nov 02 08:16:01.523410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:01.523382    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb409633dd8144b61db8bfdaac73567b761dcaa9a06adb9e6679a6fc24aa9a9c"
Nov 02 08:16:01.523622 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:01.523184    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"
Nov 02 08:16:05.553875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.553840    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"]
Nov 02 08:16:05.554933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.554909    4898 cpu_manager.go:414] "containerMap: removing container" podUID="8fa104e2-5087-434a-ac18-459608d49e6f" containerName="revoke-keys"
Nov 02 08:16:05.555021 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.554937    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa104e2-5087-434a-ac18-459608d49e6f" containerName="revoke-keys"
Nov 02 08:16:05.555021 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.554977    4898 cpu_manager.go:414] "containerMap: removing container" podUID="513e3508-7224-4fc1-ad0c-13db184d0934" containerName="revoke-keys"
Nov 02 08:16:05.555021 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.554986    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513e3508-7224-4fc1-ad0c-13db184d0934" containerName="revoke-keys"
Nov 02 08:16:05.555197 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.555155    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="513e3508-7224-4fc1-ad0c-13db184d0934" containerName="revoke-keys"
Nov 02 08:16:05.555197 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.555185    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa104e2-5087-434a-ac18-459608d49e6f" containerName="revoke-keys"
Nov 02 08:16:05.559351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.559332    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:05.575189 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.575163    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:16:05.577831 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.577806    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"]
Nov 02 08:16:05.583462 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.583429    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:16:05.584026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.583992    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:16:05.584121 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.584097    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:16:05.723455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.723419    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt6kk\" (UniqueName: \"kubernetes.io/projected/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-kube-api-access-zt6kk\") pod \"maas-api-revoke-sub-9dac6077-5kvv4\" (UID: \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:05.723455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.723456    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dac6077-5kvv4\" (UID: \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:05.723733 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.723483    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dac6077-5kvv4\" (UID: \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:05.825353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.825262    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6kk\" (UniqueName: \"kubernetes.io/projected/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-kube-api-access-zt6kk\") pod \"maas-api-revoke-sub-9dac6077-5kvv4\" (UID: \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:05.825353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.825299    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dac6077-5kvv4\" (UID: \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:05.825353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.825329    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dac6077-5kvv4\" (UID: \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:05.825973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.825948    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dac6077-5kvv4\" (UID: \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:05.827769 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.827744    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dac6077-5kvv4\" (UID: \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:05.848298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.848269    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6kk\" (UniqueName: \"kubernetes.io/projected/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-kube-api-access-zt6kk\") pod \"maas-api-revoke-sub-9dac6077-5kvv4\" (UID: \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:05.876414 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:05.876386    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:06.186185 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:16:06.186143    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbbb25f_7059_46bc_bdaf_047b816dfd5b.slice/crio-371b77bacd89d6b5549c06aca149d7880f4c7e2b48d6e93870c7591631b860ab WatchSource:0}: Error finding container 371b77bacd89d6b5549c06aca149d7880f4c7e2b48d6e93870c7591631b860ab: Status 404 returned error can't find the container with id 371b77bacd89d6b5549c06aca149d7880f4c7e2b48d6e93870c7591631b860ab
Nov 02 08:16:06.186933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:06.186905    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"]
Nov 02 08:16:06.620381 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:06.614269    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4" event={"ID":"9dbbb25f-7059-46bc-bdaf-047b816dfd5b","Type":"ContainerStarted","Data":"371b77bacd89d6b5549c06aca149d7880f4c7e2b48d6e93870c7591631b860ab"}
Nov 02 08:16:07.626384 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:07.626344    4898 generic.go:356] "Generic (PLEG): container finished" podID="9dbbb25f-7059-46bc-bdaf-047b816dfd5b" containerID="5afcf13148017328f650adb58d839fd5d66d5e86870d65ec7ee18dd48a5aec29" exitCode=0
Nov 02 08:16:07.626827 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:07.626431    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4" event={"ID":"9dbbb25f-7059-46bc-bdaf-047b816dfd5b","Type":"ContainerDied","Data":"5afcf13148017328f650adb58d839fd5d66d5e86870d65ec7ee18dd48a5aec29"}
Nov 02 08:16:08.589947 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.589910    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"]
Nov 02 08:16:08.594790 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.594769    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:08.614660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.614621    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"]
Nov 02 08:16:08.670198 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.670150    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fnwv\" (UniqueName: \"kubernetes.io/projected/438f69a6-2769-494d-b91d-3ba742d57b0d-kube-api-access-9fnwv\") pod \"maas-api-revoke-sub-e4bfdd88-42bpc\" (UID: \"438f69a6-2769-494d-b91d-3ba742d57b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:08.670836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.670811    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4bfdd88-42bpc\" (UID: \"438f69a6-2769-494d-b91d-3ba742d57b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:08.671303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.671276    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4bfdd88-42bpc\" (UID: \"438f69a6-2769-494d-b91d-3ba742d57b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:08.773042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.773000    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4bfdd88-42bpc\" (UID: \"438f69a6-2769-494d-b91d-3ba742d57b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:08.773230 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.773196    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fnwv\" (UniqueName: \"kubernetes.io/projected/438f69a6-2769-494d-b91d-3ba742d57b0d-kube-api-access-9fnwv\") pod \"maas-api-revoke-sub-e4bfdd88-42bpc\" (UID: \"438f69a6-2769-494d-b91d-3ba742d57b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:08.773306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.773239    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4bfdd88-42bpc\" (UID: \"438f69a6-2769-494d-b91d-3ba742d57b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:08.774323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.774293    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4bfdd88-42bpc\" (UID: \"438f69a6-2769-494d-b91d-3ba742d57b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:08.776680 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.776650    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4bfdd88-42bpc\" (UID: \"438f69a6-2769-494d-b91d-3ba742d57b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:08.796768 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.796744    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fnwv\" (UniqueName: \"kubernetes.io/projected/438f69a6-2769-494d-b91d-3ba742d57b0d-kube-api-access-9fnwv\") pod \"maas-api-revoke-sub-e4bfdd88-42bpc\" (UID: \"438f69a6-2769-494d-b91d-3ba742d57b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:08.834362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.834334    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:08.936366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.936333    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:08.976006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.975973    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-openshift-ca\") pod \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\" (UID: \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\") "
Nov 02 08:16:08.976193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.976039    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-xks-ca\") pod \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\" (UID: \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\") "
Nov 02 08:16:08.976300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.976275    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-kube-api-access-zt6kk\" (UniqueName: \"kubernetes.io/projected/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-kube-api-access-zt6kk\") pod \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\" (UID: \"9dbbb25f-7059-46bc-bdaf-047b816dfd5b\") "
Nov 02 08:16:08.976467 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.976365    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-openshift-ca" pod "9dbbb25f-7059-46bc-bdaf-047b816dfd5b" (UID: "9dbbb25f-7059-46bc-bdaf-047b816dfd5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:08.976930 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.976900    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:08.978696 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.978667    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-xks-ca" pod "9dbbb25f-7059-46bc-bdaf-047b816dfd5b" (UID: "9dbbb25f-7059-46bc-bdaf-047b816dfd5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:16:08.979347 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:08.979321    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-kube-api-access-zt6kk" pod "9dbbb25f-7059-46bc-bdaf-047b816dfd5b" (UID: "9dbbb25f-7059-46bc-bdaf-047b816dfd5b"). InnerVolumeSpecName "kube-api-access-zt6kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:09.078614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:09.078577    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:09.078614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:09.078608    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt6kk\" (UniqueName: \"kubernetes.io/projected/9dbbb25f-7059-46bc-bdaf-047b816dfd5b-kube-api-access-zt6kk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:09.261355 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:16:09.261319    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438f69a6_2769_494d_b91d_3ba742d57b0d.slice/crio-d070e2bb181bcbfe34032fb7cdf8cd6998c16ef001db237514d36f6019c27c83 WatchSource:0}: Error finding container d070e2bb181bcbfe34032fb7cdf8cd6998c16ef001db237514d36f6019c27c83: Status 404 returned error can't find the container with id d070e2bb181bcbfe34032fb7cdf8cd6998c16ef001db237514d36f6019c27c83
Nov 02 08:16:09.262205 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:09.262167    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"]
Nov 02 08:16:09.654580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:09.654547    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc" event={"ID":"438f69a6-2769-494d-b91d-3ba742d57b0d","Type":"ContainerStarted","Data":"d070e2bb181bcbfe34032fb7cdf8cd6998c16ef001db237514d36f6019c27c83"}
Nov 02 08:16:09.656482 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:09.656453    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4" event={"ID":"9dbbb25f-7059-46bc-bdaf-047b816dfd5b","Type":"ContainerDied","Data":"371b77bacd89d6b5549c06aca149d7880f4c7e2b48d6e93870c7591631b860ab"}
Nov 02 08:16:09.656612 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:09.656506    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="371b77bacd89d6b5549c06aca149d7880f4c7e2b48d6e93870c7591631b860ab"
Nov 02 08:16:09.656612 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:09.656465    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"
Nov 02 08:16:10.668119 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:10.668081    4898 generic.go:356] "Generic (PLEG): container finished" podID="438f69a6-2769-494d-b91d-3ba742d57b0d" containerID="04e56efba7dcb43894935d4f6d303b69b1915a2d17ca91124150dfd9dcf4549e" exitCode=0
Nov 02 08:16:10.668540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:10.668167    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc" event={"ID":"438f69a6-2769-494d-b91d-3ba742d57b0d","Type":"ContainerDied","Data":"04e56efba7dcb43894935d4f6d303b69b1915a2d17ca91124150dfd9dcf4549e"}
Nov 02 08:16:11.883795 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:11.883765    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:12.034671 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:12.034568    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-xks-ca\") pod \"438f69a6-2769-494d-b91d-3ba742d57b0d\" (UID: \"438f69a6-2769-494d-b91d-3ba742d57b0d\") "
Nov 02 08:16:12.034851 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:12.034688    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-openshift-ca\") pod \"438f69a6-2769-494d-b91d-3ba742d57b0d\" (UID: \"438f69a6-2769-494d-b91d-3ba742d57b0d\") "
Nov 02 08:16:12.034988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:12.034967    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/438f69a6-2769-494d-b91d-3ba742d57b0d-kube-api-access-9fnwv\" (UniqueName: \"kubernetes.io/projected/438f69a6-2769-494d-b91d-3ba742d57b0d-kube-api-access-9fnwv\") pod \"438f69a6-2769-494d-b91d-3ba742d57b0d\" (UID: \"438f69a6-2769-494d-b91d-3ba742d57b0d\") "
Nov 02 08:16:12.035704 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:12.035672    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-openshift-ca" pod "438f69a6-2769-494d-b91d-3ba742d57b0d" (UID: "438f69a6-2769-494d-b91d-3ba742d57b0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:12.036379 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:12.036354    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:12.037942 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:12.037909    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438f69a6-2769-494d-b91d-3ba742d57b0d-kube-api-access-9fnwv" pod "438f69a6-2769-494d-b91d-3ba742d57b0d" (UID: "438f69a6-2769-494d-b91d-3ba742d57b0d"). InnerVolumeSpecName "kube-api-access-9fnwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:12.038425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:12.038393    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-xks-ca" pod "438f69a6-2769-494d-b91d-3ba742d57b0d" (UID: "438f69a6-2769-494d-b91d-3ba742d57b0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:16:12.138083 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:12.138048    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fnwv\" (UniqueName: \"kubernetes.io/projected/438f69a6-2769-494d-b91d-3ba742d57b0d-kube-api-access-9fnwv\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:12.138083 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:12.138079    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438f69a6-2769-494d-b91d-3ba742d57b0d-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:12.689801 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:12.689769    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"
Nov 02 08:16:12.689801 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:12.689791    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc" event={"ID":"438f69a6-2769-494d-b91d-3ba742d57b0d","Type":"ContainerDied","Data":"d070e2bb181bcbfe34032fb7cdf8cd6998c16ef001db237514d36f6019c27c83"}
Nov 02 08:16:12.690059 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:12.689881    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d070e2bb181bcbfe34032fb7cdf8cd6998c16ef001db237514d36f6019c27c83"
Nov 02 08:16:14.025684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:14.025651    4898 scope.go:122] "RemoveContainer" containerID="637b6fb437ff275e0f2cfcd6b5fe8a9a12df293e37214b20b2a0ea6f1421e05c"
Nov 02 08:16:14.038038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:14.038019    4898 scope.go:122] "RemoveContainer" containerID="79b30f3d15f5a290e2bb13e243ab86b8efc7de5577278994a5c94d844ac67168"
Nov 02 08:16:14.048334 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:14.048310    4898 scope.go:122] "RemoveContainer" containerID="5c24f975df9f60a011361d4e21014927870aecd6027fe115f4e5305391f73d67"
Nov 02 08:16:14.058177 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:14.058151    4898 scope.go:122] "RemoveContainer" containerID="ab828990841a70be5255bbf51297f2198bc5ef55b6c1e1c2be5533c44ed5b7ad"
Nov 02 08:16:14.994403 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:16:14.994363    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:16:17.771555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.766420    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"]
Nov 02 08:16:17.771555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.767694    4898 cpu_manager.go:414] "containerMap: removing container" podUID="438f69a6-2769-494d-b91d-3ba742d57b0d" containerName="revoke-keys"
Nov 02 08:16:17.771555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.767717    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="438f69a6-2769-494d-b91d-3ba742d57b0d" containerName="revoke-keys"
Nov 02 08:16:17.771555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.767747    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9dbbb25f-7059-46bc-bdaf-047b816dfd5b" containerName="revoke-keys"
Nov 02 08:16:17.771555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.767755    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbbb25f-7059-46bc-bdaf-047b816dfd5b" containerName="revoke-keys"
Nov 02 08:16:17.771555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.767947    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="438f69a6-2769-494d-b91d-3ba742d57b0d" containerName="revoke-keys"
Nov 02 08:16:17.771555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.767963    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbbb25f-7059-46bc-bdaf-047b816dfd5b" containerName="revoke-keys"
Nov 02 08:16:17.776522 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.774701    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.776694 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.776611    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"]
Nov 02 08:16:17.790488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.790270    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"agc-cred-f45c1914cd99\""
Nov 02 08:16:17.790488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.790338    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-e2e--71cd8499-dockercfg-hftbt\""
Nov 02 08:16:17.790488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.790400    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"openshift-service-ca.crt\""
Nov 02 08:16:17.802089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.802064    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"routing-overlay\""
Nov 02 08:16:17.802213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.802142    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-736f7c\""
Nov 02 08:16:17.802294 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.802213    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-736f7c\"/\"kube-root-ca.crt\""
Nov 02 08:16:17.837878 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.837845    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.838033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.838011    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9r68\" (UniqueName: \"kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-kube-api-access-p9r68\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.838158 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.838140    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.838282 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.838256    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.939850 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.939816    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.940031 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.939928    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9r68\" (UniqueName: \"kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-kube-api-access-p9r68\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.940081 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.940035    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.940156 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.940121    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.940613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.940582    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.940726 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.940661    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.942616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.942588    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.966246 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.966221    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9r68\" (UniqueName: \"kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-kube-api-access-p9r68\") pod \"payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") " pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:17.975804 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.975778    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"]
Nov 02 08:16:17.981183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.981164    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:17.996617 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.996585    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:16:17.996751 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.996558    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:16:17.996751 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:17.996673    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:16:18.000333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.000312    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"]
Nov 02 08:16:18.007561 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.007140    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:16:18.122963 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.115993    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:18.294914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.270801    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4d35571-6hkkl\" (UID: \"68463862-7d8c-468c-b22a-14783ed38fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:18.294914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.287030    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4d35571-6hkkl\" (UID: \"68463862-7d8c-468c-b22a-14783ed38fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:18.294914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.287171    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smfk8\" (UniqueName: \"kubernetes.io/projected/68463862-7d8c-468c-b22a-14783ed38fe3-kube-api-access-smfk8\") pod \"maas-api-revoke-sub-b4d35571-6hkkl\" (UID: \"68463862-7d8c-468c-b22a-14783ed38fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:18.389277 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.389168    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4d35571-6hkkl\" (UID: \"68463862-7d8c-468c-b22a-14783ed38fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:18.389277 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.389236    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4d35571-6hkkl\" (UID: \"68463862-7d8c-468c-b22a-14783ed38fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:18.389277 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.389277    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smfk8\" (UniqueName: \"kubernetes.io/projected/68463862-7d8c-468c-b22a-14783ed38fe3-kube-api-access-smfk8\") pod \"maas-api-revoke-sub-b4d35571-6hkkl\" (UID: \"68463862-7d8c-468c-b22a-14783ed38fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:18.392816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.390040    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4d35571-6hkkl\" (UID: \"68463862-7d8c-468c-b22a-14783ed38fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:18.394250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.394220    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4d35571-6hkkl\" (UID: \"68463862-7d8c-468c-b22a-14783ed38fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:18.561912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.559256    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smfk8\" (UniqueName: \"kubernetes.io/projected/68463862-7d8c-468c-b22a-14783ed38fe3-kube-api-access-smfk8\") pod \"maas-api-revoke-sub-b4d35571-6hkkl\" (UID: \"68463862-7d8c-468c-b22a-14783ed38fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:18.623914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.614141    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:18.674377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.658937    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"]
Nov 02 08:16:18.710918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.706924    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"]
Nov 02 08:16:18.718711 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.715252    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:18.730620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.730550    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"agc-cred-a89b1ea64886\""
Nov 02 08:16:18.730789 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.730699    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-e2e--1a3816b7-dockercfg-52g6v\""
Nov 02 08:16:18.732122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.732097    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"]
Nov 02 08:16:18.732541 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:16:18.732486    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a950de_4935_42e3_a810_bc8987c3740d.slice/crio-d5aae47434cb3bf56c2f0998fa330f974dce5e3bbdf8dbe34af17a9379324339 WatchSource:0}: Error finding container d5aae47434cb3bf56c2f0998fa330f974dce5e3bbdf8dbe34af17a9379324339: Status 404 returned error can't find the container with id d5aae47434cb3bf56c2f0998fa330f974dce5e3bbdf8dbe34af17a9379324339
Nov 02 08:16:18.732663 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.732567    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"openshift-service-ca.crt\""
Nov 02 08:16:18.734728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.734708    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"routing-overlay\""
Nov 02 08:16:18.737115 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.734931    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"kube-root-ca.crt\""
Nov 02 08:16:18.737115 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.735099    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2dde84\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2dde84\""
Nov 02 08:16:18.782341 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.782244    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22" event={"ID":"07a950de-4935-42e3-a810-bc8987c3740d","Type":"ContainerStarted","Data":"d5aae47434cb3bf56c2f0998fa330f974dce5e3bbdf8dbe34af17a9379324339"}
Nov 02 08:16:18.855677 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.855271    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:18.855677 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.855626    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:18.855883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.855677    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:18.855883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.855715    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ssfw\" (UniqueName: \"kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-kube-api-access-4ssfw\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:18.957598 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.957496    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:18.957598 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.957560    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:18.957598 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.957591    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ssfw\" (UniqueName: \"kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-kube-api-access-4ssfw\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:18.957981 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.957682    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:18.958429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.958402    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:18.958727 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.958679    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:18.961166 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.961135    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:18.983056 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:18.983027    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ssfw\" (UniqueName: \"kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-kube-api-access-4ssfw\") pod \"payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") " pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:19.045468 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:16:19.045438    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68463862_7d8c_468c_b22a_14783ed38fe3.slice/crio-291b8501c8e37bad649a889e33a8e6b544ef0f804d687c218f3290f4ca7acc34 WatchSource:0}: Error finding container 291b8501c8e37bad649a889e33a8e6b544ef0f804d687c218f3290f4ca7acc34: Status 404 returned error can't find the container with id 291b8501c8e37bad649a889e33a8e6b544ef0f804d687c218f3290f4ca7acc34
Nov 02 08:16:19.046112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.046089    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"]
Nov 02 08:16:19.047864 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.047845    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:19.369773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.369736    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"]
Nov 02 08:16:19.794433 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.794340    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9" event={"ID":"9fda0c98-392b-4858-b29e-f0b87ff6e719","Type":"ContainerStarted","Data":"25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8"}
Nov 02 08:16:19.794433 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.794412    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:19.794924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.794440    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9" event={"ID":"9fda0c98-392b-4858-b29e-f0b87ff6e719","Type":"ContainerStarted","Data":"df32531b5bc622fe206747c7158757027026fd1b21ef9dc952c63041ef9163c4"}
Nov 02 08:16:19.797036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.797004    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22" event={"ID":"07a950de-4935-42e3-a810-bc8987c3740d","Type":"ContainerStarted","Data":"c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3"}
Nov 02 08:16:19.797169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.797091    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:19.799098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.799070    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl" event={"ID":"68463862-7d8c-468c-b22a-14783ed38fe3","Type":"ContainerStarted","Data":"4d888e4dae980ea3e79cd70082333476e6807aa29d040cc8d03f1f214c6c377f"}
Nov 02 08:16:19.799195 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.799116    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl" event={"ID":"68463862-7d8c-468c-b22a-14783ed38fe3","Type":"ContainerStarted","Data":"291b8501c8e37bad649a889e33a8e6b544ef0f804d687c218f3290f4ca7acc34"}
Nov 02 08:16:19.838876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.838818    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9" podStartSLOduration=1.838805893 podStartE2EDuration="1.838805893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:19.835235086 +0000 UTC m=+3130.525562051" watchObservedRunningTime="2026-11-02 08:16:19.838805893 +0000 UTC m=+3130.529132853"
Nov 02 08:16:19.875716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.875666    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl" podStartSLOduration=2.875653949 podStartE2EDuration="2.875653949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:19.874863887 +0000 UTC m=+3130.565190852" watchObservedRunningTime="2026-11-02 08:16:19.875653949 +0000 UTC m=+3130.565980913"
Nov 02 08:16:19.922631 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:19.922577    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22" podStartSLOduration=2.922563576 podStartE2EDuration="2.922563576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:19.92181964 +0000 UTC m=+3130.612146605" watchObservedRunningTime="2026-11-02 08:16:19.922563576 +0000 UTC m=+3130.612890540"
Nov 02 08:16:20.810495 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:20.810453    4898 generic.go:356] "Generic (PLEG): container finished" podID="68463862-7d8c-468c-b22a-14783ed38fe3" containerID="4d888e4dae980ea3e79cd70082333476e6807aa29d040cc8d03f1f214c6c377f" exitCode=0
Nov 02 08:16:20.810961 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:20.810533    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl" event={"ID":"68463862-7d8c-468c-b22a-14783ed38fe3","Type":"ContainerDied","Data":"4d888e4dae980ea3e79cd70082333476e6807aa29d040cc8d03f1f214c6c377f"}
Nov 02 08:16:22.021841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.021813    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:22.171266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.171235    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-openshift-ca\") pod \"68463862-7d8c-468c-b22a-14783ed38fe3\" (UID: \"68463862-7d8c-468c-b22a-14783ed38fe3\") "
Nov 02 08:16:22.171477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.171437    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68463862-7d8c-468c-b22a-14783ed38fe3-kube-api-access-smfk8\" (UniqueName: \"kubernetes.io/projected/68463862-7d8c-468c-b22a-14783ed38fe3-kube-api-access-smfk8\") pod \"68463862-7d8c-468c-b22a-14783ed38fe3\" (UID: \"68463862-7d8c-468c-b22a-14783ed38fe3\") "
Nov 02 08:16:22.171590 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.171573    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-xks-ca\") pod \"68463862-7d8c-468c-b22a-14783ed38fe3\" (UID: \"68463862-7d8c-468c-b22a-14783ed38fe3\") "
Nov 02 08:16:22.171695 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.171670    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-openshift-ca" pod "68463862-7d8c-468c-b22a-14783ed38fe3" (UID: "68463862-7d8c-468c-b22a-14783ed38fe3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:22.172256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.172233    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:22.174268 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.174241    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-xks-ca" pod "68463862-7d8c-468c-b22a-14783ed38fe3" (UID: "68463862-7d8c-468c-b22a-14783ed38fe3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:16:22.174365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.174288    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68463862-7d8c-468c-b22a-14783ed38fe3-kube-api-access-smfk8" pod "68463862-7d8c-468c-b22a-14783ed38fe3" (UID: "68463862-7d8c-468c-b22a-14783ed38fe3"). InnerVolumeSpecName "kube-api-access-smfk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:22.273780 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.273744    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68463862-7d8c-468c-b22a-14783ed38fe3-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:22.273986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.273784    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smfk8\" (UniqueName: \"kubernetes.io/projected/68463862-7d8c-468c-b22a-14783ed38fe3-kube-api-access-smfk8\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:22.835911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.832968    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"
Nov 02 08:16:22.835911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.833000    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl" event={"ID":"68463862-7d8c-468c-b22a-14783ed38fe3","Type":"ContainerDied","Data":"291b8501c8e37bad649a889e33a8e6b544ef0f804d687c218f3290f4ca7acc34"}
Nov 02 08:16:22.835911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:22.833052    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="291b8501c8e37bad649a889e33a8e6b544ef0f804d687c218f3290f4ca7acc34"
Nov 02 08:16:28.992950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:28.989843    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"]
Nov 02 08:16:28.995650 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:28.995621    4898 cpu_manager.go:414] "containerMap: removing container" podUID="68463862-7d8c-468c-b22a-14783ed38fe3" containerName="revoke-keys"
Nov 02 08:16:28.995855 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:28.995839    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68463862-7d8c-468c-b22a-14783ed38fe3" containerName="revoke-keys"
Nov 02 08:16:28.996398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:28.996378    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="68463862-7d8c-468c-b22a-14783ed38fe3" containerName="revoke-keys"
Nov 02 08:16:29.019958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.012850    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"]
Nov 02 08:16:29.019958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.013005    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:29.037342 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.035339    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:16:29.037342 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.035447    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:16:29.037342 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.035502    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:16:29.037342 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.035687    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:16:29.289218 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.287832    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf093132-7gr5m\" (UID: \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:29.289218 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.288032    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ch4\" (UniqueName: \"kubernetes.io/projected/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-kube-api-access-q2ch4\") pod \"maas-api-revoke-sub-cf093132-7gr5m\" (UID: \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:29.289218 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.288255    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf093132-7gr5m\" (UID: \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:29.390149 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.390112    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf093132-7gr5m\" (UID: \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:29.390339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.390245    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf093132-7gr5m\" (UID: \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:29.390391 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.390350    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ch4\" (UniqueName: \"kubernetes.io/projected/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-kube-api-access-q2ch4\") pod \"maas-api-revoke-sub-cf093132-7gr5m\" (UID: \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:29.390885 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.390858    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf093132-7gr5m\" (UID: \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:29.392784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.392754    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf093132-7gr5m\" (UID: \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:29.414040 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.414014    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ch4\" (UniqueName: \"kubernetes.io/projected/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-kube-api-access-q2ch4\") pod \"maas-api-revoke-sub-cf093132-7gr5m\" (UID: \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:29.641806 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:29.637968    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:30.011401 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:16:30.010800    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3684e5_9c55_4ae6_9b49_6e48ade8686e.slice/crio-d2097149c452f1940c680e59bc85cbdf90edc82b753715578851f3043e7edca4 WatchSource:0}: Error finding container d2097149c452f1940c680e59bc85cbdf90edc82b753715578851f3043e7edca4: Status 404 returned error can't find the container with id d2097149c452f1940c680e59bc85cbdf90edc82b753715578851f3043e7edca4
Nov 02 08:16:30.014361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:30.013999    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"]
Nov 02 08:16:30.322394 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:30.322307    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m" event={"ID":"6a3684e5-9c55-4ae6-9b49-6e48ade8686e","Type":"ContainerStarted","Data":"d2097149c452f1940c680e59bc85cbdf90edc82b753715578851f3043e7edca4"}
Nov 02 08:16:30.814114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:30.814079    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:30.814326 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:30.814244    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:31.591921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:31.568864    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m" event={"ID":"6a3684e5-9c55-4ae6-9b49-6e48ade8686e","Type":"ContainerStarted","Data":"8af6351d26962b9654d028842bd87f18b65e75ace524849183dbeed0012f8aff"}
Nov 02 08:16:32.590269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:32.590228    4898 generic.go:356] "Generic (PLEG): container finished" podID="6a3684e5-9c55-4ae6-9b49-6e48ade8686e" containerID="8af6351d26962b9654d028842bd87f18b65e75ace524849183dbeed0012f8aff" exitCode=0
Nov 02 08:16:32.590434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:32.590317    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m" event={"ID":"6a3684e5-9c55-4ae6-9b49-6e48ade8686e","Type":"ContainerDied","Data":"8af6351d26962b9654d028842bd87f18b65e75ace524849183dbeed0012f8aff"}
Nov 02 08:16:33.977951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:33.960552    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:34.052929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:34.052850    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-openshift-ca\") pod \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\" (UID: \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\") "
Nov 02 08:16:34.053228 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:34.053203    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-openshift-ca" pod "6a3684e5-9c55-4ae6-9b49-6e48ade8686e" (UID: "6a3684e5-9c55-4ae6-9b49-6e48ade8686e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:34.053360 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:34.053283    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-kube-api-access-q2ch4\" (UniqueName: \"kubernetes.io/projected/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-kube-api-access-q2ch4\") pod \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\" (UID: \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\") "
Nov 02 08:16:34.053360 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:34.053340    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-xks-ca\") pod \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\" (UID: \"6a3684e5-9c55-4ae6-9b49-6e48ade8686e\") "
Nov 02 08:16:34.054007 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:34.053984    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:34.078973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:34.074047    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-xks-ca" pod "6a3684e5-9c55-4ae6-9b49-6e48ade8686e" (UID: "6a3684e5-9c55-4ae6-9b49-6e48ade8686e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:16:34.078973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:34.074193    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-kube-api-access-q2ch4" pod "6a3684e5-9c55-4ae6-9b49-6e48ade8686e" (UID: "6a3684e5-9c55-4ae6-9b49-6e48ade8686e"). InnerVolumeSpecName "kube-api-access-q2ch4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:34.162913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:34.156178    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2ch4\" (UniqueName: \"kubernetes.io/projected/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-kube-api-access-q2ch4\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:34.162913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:34.156221    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3684e5-9c55-4ae6-9b49-6e48ade8686e-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:34.614694 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:34.614651    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"
Nov 02 08:16:34.614694 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:34.614672    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m" event={"ID":"6a3684e5-9c55-4ae6-9b49-6e48ade8686e","Type":"ContainerDied","Data":"d2097149c452f1940c680e59bc85cbdf90edc82b753715578851f3043e7edca4"}
Nov 02 08:16:34.614940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:34.614740    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2097149c452f1940c680e59bc85cbdf90edc82b753715578851f3043e7edca4"
Nov 02 08:16:39.678584 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:39.678546    4898 generic.go:356] "Generic (PLEG): container finished" podID="cb263662-7712-4909-92c9-deb40f800844" containerID="d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6" exitCode=0
Nov 02 08:16:39.679166 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:39.678589    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" event={"ID":"cb263662-7712-4909-92c9-deb40f800844","Type":"ContainerDied","Data":"d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6"}
Nov 02 08:16:39.679343 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:39.679322    4898 scope.go:122] "RemoveContainer" containerID="d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6"
Nov 02 08:16:39.682097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:39.682060    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:16:39.682223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:39.682201    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:16:39.682272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:39.682262    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:16:40.670759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:40.670719    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"]
Nov 02 08:16:40.690640 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:40.690609    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" event={"ID":"cb263662-7712-4909-92c9-deb40f800844","Type":"ContainerStarted","Data":"4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14"}
Nov 02 08:16:41.202868 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:41.201233    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:16:41.203612 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:41.203181    4898 patch_prober.go:29] interesting pod/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Nov 02 08:16:41.203612 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:41.203238    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Nov 02 08:16:42.200365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:42.200320    4898 patch_prober.go:29] interesting pod/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Nov 02 08:16:42.201137 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:42.201095    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Nov 02 08:16:42.428112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:42.428071    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"]
Nov 02 08:16:43.039912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:43.031639    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:16:43.198456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:43.198415    4898 patch_prober.go:29] interesting pod/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Nov 02 08:16:43.198668 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:43.198476    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Nov 02 08:16:44.198269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:44.198219    4898 patch_prober.go:29] interesting pod/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Nov 02 08:16:44.198787 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:44.198281    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Nov 02 08:16:45.020324 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:16:45.019531    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:16:45.198812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:45.198773    4898 patch_prober.go:29] interesting pod/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Nov 02 08:16:45.199332 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:45.198841    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Nov 02 08:16:45.687013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:45.686970    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22" podUID="07a950de-4935-42e3-a810-bc8987c3740d" containerName="payload-processing" containerID="cri-o://c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3" gracePeriod=25
Nov 02 08:16:46.042321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.042293    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:46.120754 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.120718    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-kube-api-access-p9r68\" (UniqueName: \"kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-kube-api-access-p9r68\") pod \"07a950de-4935-42e3-a810-bc8987c3740d\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") "
Nov 02 08:16:46.120991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.120835    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-plugins-config-volume\") pod \"07a950de-4935-42e3-a810-bc8987c3740d\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") "
Nov 02 08:16:46.120991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.120879    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-provider-credentials\") pod \"07a950de-4935-42e3-a810-bc8987c3740d\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") "
Nov 02 08:16:46.121125 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.121052    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-routing-overlay-volume\") pod \"07a950de-4935-42e3-a810-bc8987c3740d\" (UID: \"07a950de-4935-42e3-a810-bc8987c3740d\") "
Nov 02 08:16:46.121480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.121452    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-plugins-config-volume" pod "07a950de-4935-42e3-a810-bc8987c3740d" (UID: "07a950de-4935-42e3-a810-bc8987c3740d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:46.121772 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.121751    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:46.122167 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.122142    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-routing-overlay-volume" pod "07a950de-4935-42e3-a810-bc8987c3740d" (UID: "07a950de-4935-42e3-a810-bc8987c3740d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:46.123796 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.123771    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-provider-credentials" pod "07a950de-4935-42e3-a810-bc8987c3740d" (UID: "07a950de-4935-42e3-a810-bc8987c3740d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:46.124065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.124044    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-kube-api-access-p9r68" pod "07a950de-4935-42e3-a810-bc8987c3740d" (UID: "07a950de-4935-42e3-a810-bc8987c3740d"). InnerVolumeSpecName "kube-api-access-p9r68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:46.198222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.198172    4898 patch_prober.go:29] interesting pod/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body=
Nov 02 08:16:46.198437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.198235    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused"
Nov 02 08:16:46.223540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.223500    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9r68\" (UniqueName: \"kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-kube-api-access-p9r68\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:46.223540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.223533    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a950de-4935-42e3-a810-bc8987c3740d-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:46.223540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.223545    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a950de-4935-42e3-a810-bc8987c3740d-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:46.766724 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.766679    4898 generic.go:356] "Generic (PLEG): container finished" podID="07a950de-4935-42e3-a810-bc8987c3740d" containerID="c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3" exitCode=0
Nov 02 08:16:46.766964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.766946    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22" event={"ID":"07a950de-4935-42e3-a810-bc8987c3740d","Type":"ContainerDied","Data":"c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3"}
Nov 02 08:16:46.767085 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.767001    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22" event={"ID":"07a950de-4935-42e3-a810-bc8987c3740d","Type":"ContainerDied","Data":"d5aae47434cb3bf56c2f0998fa330f974dce5e3bbdf8dbe34af17a9379324339"}
Nov 02 08:16:46.767085 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.767031    4898 scope.go:122] "RemoveContainer" containerID="c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3"
Nov 02 08:16:46.767220 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.767202    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"
Nov 02 08:16:46.786512 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.786477    4898 scope.go:122] "RemoveContainer" containerID="c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3"
Nov 02 08:16:46.786863 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:16:46.786836    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3\": container with ID starting with c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3 not found: ID does not exist" containerID="c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3"
Nov 02 08:16:46.787033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.786873    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3"} err="failed to get container status \"c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3\": rpc error: code = NotFound desc = could not find container \"c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3\": container with ID starting with c653c1ae8ba6e8555f940686e074485101a2900a862f78956843791c45fa9bf3 not found: ID does not exist"
Nov 02 08:16:46.876910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.875055    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"]
Nov 02 08:16:46.885879 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:46.885837    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-736f7c/payload-processing-external-model-e2e-extmod-a-736f7c-cdf7nsh22"]
Nov 02 08:16:47.203459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.203428    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:16:47.204637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.204611    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:16:47.428641 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.428573    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9" podUID="9fda0c98-392b-4858-b29e-f0b87ff6e719" containerName="payload-processing" containerID="cri-o://25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8" gracePeriod=25
Nov 02 08:16:47.469727 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.469691    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a950de-4935-42e3-a810-bc8987c3740d" path="/var/lib/kubelet/pods/07a950de-4935-42e3-a810-bc8987c3740d/volumes"
Nov 02 08:16:47.764341 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.764311    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:47.779066 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.779035    4898 generic.go:356] "Generic (PLEG): container finished" podID="9fda0c98-392b-4858-b29e-f0b87ff6e719" containerID="25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8" exitCode=0
Nov 02 08:16:47.779212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.779080    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"
Nov 02 08:16:47.779212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.779127    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9" event={"ID":"9fda0c98-392b-4858-b29e-f0b87ff6e719","Type":"ContainerDied","Data":"25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8"}
Nov 02 08:16:47.779212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.779188    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9" event={"ID":"9fda0c98-392b-4858-b29e-f0b87ff6e719","Type":"ContainerDied","Data":"df32531b5bc622fe206747c7158757027026fd1b21ef9dc952c63041ef9163c4"}
Nov 02 08:16:47.779364 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.779221    4898 scope.go:122] "RemoveContainer" containerID="25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8"
Nov 02 08:16:47.790999 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.790983    4898 scope.go:122] "RemoveContainer" containerID="25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8"
Nov 02 08:16:47.791332 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:16:47.791313    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8\": container with ID starting with 25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8 not found: ID does not exist" containerID="25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8"
Nov 02 08:16:47.791429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.791340    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8"} err="failed to get container status \"25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8\": rpc error: code = NotFound desc = could not find container \"25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8\": container with ID starting with 25c87620e58528b5bc863bfccc646ce0cfd4f34e8718b183cb1fb3e8e912ffc8 not found: ID does not exist"
Nov 02 08:16:47.857555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.857518    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-provider-credentials\") pod \"9fda0c98-392b-4858-b29e-f0b87ff6e719\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") "
Nov 02 08:16:47.858029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.857997    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-routing-overlay-volume\") pod \"9fda0c98-392b-4858-b29e-f0b87ff6e719\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") "
Nov 02 08:16:47.864122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.858403    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-routing-overlay-volume" pod "9fda0c98-392b-4858-b29e-f0b87ff6e719" (UID: "9fda0c98-392b-4858-b29e-f0b87ff6e719"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:47.864122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.859467    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-kube-api-access-4ssfw\" (UniqueName: \"kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-kube-api-access-4ssfw\") pod \"9fda0c98-392b-4858-b29e-f0b87ff6e719\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") "
Nov 02 08:16:47.864122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.859611    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-plugins-config-volume\") pod \"9fda0c98-392b-4858-b29e-f0b87ff6e719\" (UID: \"9fda0c98-392b-4858-b29e-f0b87ff6e719\") "
Nov 02 08:16:47.864122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.860506    4898 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-routing-overlay-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:47.864122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.861671    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-plugins-config-volume" pod "9fda0c98-392b-4858-b29e-f0b87ff6e719" (UID: "9fda0c98-392b-4858-b29e-f0b87ff6e719"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:16:47.864122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.862185    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-provider-credentials" pod "9fda0c98-392b-4858-b29e-f0b87ff6e719" (UID: "9fda0c98-392b-4858-b29e-f0b87ff6e719"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:47.865561 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.865532    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-kube-api-access-4ssfw" pod "9fda0c98-392b-4858-b29e-f0b87ff6e719" (UID: "9fda0c98-392b-4858-b29e-f0b87ff6e719"). InnerVolumeSpecName "kube-api-access-4ssfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:16:47.961981 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.961953    4898 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-provider-credentials\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:47.961981 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.961979    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ssfw\" (UniqueName: \"kubernetes.io/projected/9fda0c98-392b-4858-b29e-f0b87ff6e719-kube-api-access-4ssfw\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:47.962218 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:47.961990    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fda0c98-392b-4858-b29e-f0b87ff6e719-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:16:48.130809 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.130770    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"]
Nov 02 08:16:48.142932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.142906    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2dde84/payload-processing-external-model-e2e-extmod-b-2dde84-7bdf5fww9"]
Nov 02 08:16:48.303139 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.303097    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"]
Nov 02 08:16:48.303937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.303919    4898 cpu_manager.go:414] "containerMap: removing container" podUID="07a950de-4935-42e3-a810-bc8987c3740d" containerName="payload-processing"
Nov 02 08:16:48.303937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.303936    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a950de-4935-42e3-a810-bc8987c3740d" containerName="payload-processing"
Nov 02 08:16:48.304108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.303947    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6a3684e5-9c55-4ae6-9b49-6e48ade8686e" containerName="revoke-keys"
Nov 02 08:16:48.304108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.303953    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3684e5-9c55-4ae6-9b49-6e48ade8686e" containerName="revoke-keys"
Nov 02 08:16:48.304108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.303977    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9fda0c98-392b-4858-b29e-f0b87ff6e719" containerName="payload-processing"
Nov 02 08:16:48.304108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.303984    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fda0c98-392b-4858-b29e-f0b87ff6e719" containerName="payload-processing"
Nov 02 08:16:48.304343 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.304140    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a950de-4935-42e3-a810-bc8987c3740d" containerName="payload-processing"
Nov 02 08:16:48.304343 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.304158    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3684e5-9c55-4ae6-9b49-6e48ade8686e" containerName="revoke-keys"
Nov 02 08:16:48.304343 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.304166    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fda0c98-392b-4858-b29e-f0b87ff6e719" containerName="payload-processing"
Nov 02 08:16:48.309779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.309755    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.327864 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.327838    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b72fa3-openshift-default-dockercfg-s6qk6\""
Nov 02 08:16:48.331242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.331219    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"]
Nov 02 08:16:48.367741 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.367655    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9341cd-5407-4907-a11d-b3af85a6baa5-istiod-ca-cert\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.367741 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.367730    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-podinfo\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.367992 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.367870    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-socket\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.367992 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.367937    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-data\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.368111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.368003    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-envoy\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.368111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.368031    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-certs\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.368111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.368067    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-credential-socket\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.368111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.368094    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-token\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.368350 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.368194    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2494k\" (UniqueName: \"kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-kube-api-access-2494k\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.483916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.483396    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-podinfo\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.483916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.483497    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-socket\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.483916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.483607    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-data\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.483916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.483679    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-envoy\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.483916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.483710    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-certs\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.483916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.483765    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-credential-socket\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.483916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.483792    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-token\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.483916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.483904    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2494k\" (UniqueName: \"kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-kube-api-access-2494k\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.484837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.484120    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9341cd-5407-4907-a11d-b3af85a6baa5-istiod-ca-cert\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.493908 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.485076    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9341cd-5407-4907-a11d-b3af85a6baa5-istiod-ca-cert\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.493908 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.485812    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-certs\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.507776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.502054    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-data\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.507776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.502648    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-credential-socket\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.531912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.526233    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-envoy\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.531912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.526503    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-socket\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.683908 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.673589    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-token\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.737910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.684753    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-podinfo\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.746039 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.689527    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2494k\" (UniqueName: \"kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-kube-api-access-2494k\") pod \"e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") " pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:48.939405 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:48.938285    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:49.312001 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:16:49.311969    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9341cd_5407_4907_a11d_b3af85a6baa5.slice/crio-ee0a2f7b7f2b030fc5adc14dbba5be9578cf71156421f96cb1f63800a7fd0f8f WatchSource:0}: Error finding container ee0a2f7b7f2b030fc5adc14dbba5be9578cf71156421f96cb1f63800a7fd0f8f: Status 404 returned error can't find the container with id ee0a2f7b7f2b030fc5adc14dbba5be9578cf71156421f96cb1f63800a7fd0f8f
Nov 02 08:16:49.314602 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:49.314578    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"]
Nov 02 08:16:49.315437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:49.315402    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:16:49.315499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:49.315467    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:16:49.315548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:49.315498    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:16:49.471297 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:49.471261    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fda0c98-392b-4858-b29e-f0b87ff6e719" path="/var/lib/kubelet/pods/9fda0c98-392b-4858-b29e-f0b87ff6e719/volumes"
Nov 02 08:16:49.960435 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:49.960394    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" event={"ID":"bf9341cd-5407-4907-a11d-b3af85a6baa5","Type":"ContainerStarted","Data":"681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830"}
Nov 02 08:16:49.960976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:49.960462    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" event={"ID":"bf9341cd-5407-4907-a11d-b3af85a6baa5","Type":"ContainerStarted","Data":"ee0a2f7b7f2b030fc5adc14dbba5be9578cf71156421f96cb1f63800a7fd0f8f"}
Nov 02 08:16:50.021146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:50.021093    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" podStartSLOduration=2.021078562 podStartE2EDuration="2.021078562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:16:50.017691906 +0000 UTC m=+3160.708018918" watchObservedRunningTime="2026-11-02 08:16:50.021078562 +0000 UTC m=+3160.711405595"
Nov 02 08:16:50.938900 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:50.938851    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:50.945464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:50.945439    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:50.972952 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:50.972922    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:16:50.973976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:16:50.973952    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:17:01.033875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:01.033835    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"]
Nov 02 08:17:01.045589 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:01.045560    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebc53711-tf5rr"]
Nov 02 08:17:01.467678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:01.467643    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb5c044-7cf5-45b3-ac2d-28ced2e72686" path="/var/lib/kubelet/pods/ddb5c044-7cf5-45b3-ac2d-28ced2e72686/volumes"
Nov 02 08:17:04.036293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:04.036255    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"]
Nov 02 08:17:04.048047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:04.048016    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d3a354-zjj7t"]
Nov 02 08:17:05.469904 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:05.469853    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3b9f44-fd12-4500-84cd-76fbf8b0e20a" path="/var/lib/kubelet/pods/1d3b9f44-fd12-4500-84cd-76fbf8b0e20a/volumes"
Nov 02 08:17:15.045169 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:17:15.045092    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:17:22.295160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:22.295068    4898 patch_prober.go:29] interesting pod/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:17:22.295160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:22.295139    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:17:26.428510 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.428473    4898 patch_prober.go:29] interesting pod/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Nov 02 08:17:26.428989 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.428536    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Nov 02 08:17:26.577990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.577956    4898 generic.go:356] "Generic (PLEG): container finished" podID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerID="4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37" exitCode=0
Nov 02 08:17:26.578193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.578028    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" event={"ID":"ed94547d-f04f-4401-a356-fd38ffdcf2e4","Type":"ContainerDied","Data":"4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37"}
Nov 02 08:17:26.578660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.578632    4898 scope.go:122] "RemoveContainer" containerID="4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37"
Nov 02 08:17:26.580621 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.580595    4898 generic.go:356] "Generic (PLEG): container finished" podID="1900eb00-365d-401d-a399-946fb0c03bfa" containerID="f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c" exitCode=0
Nov 02 08:17:26.580734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.580637    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" event={"ID":"1900eb00-365d-401d-a399-946fb0c03bfa","Type":"ContainerDied","Data":"f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c"}
Nov 02 08:17:26.581093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.581072    4898 scope.go:122] "RemoveContainer" containerID="f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c"
Nov 02 08:17:26.581238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.581205    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:17:26.581313 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.581295    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:17:26.581372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.581352    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:17:26.582816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.582782    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:17:26.582938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.582848    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:17:26.582938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:26.582880    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:17:27.597143 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:27.597082    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" event={"ID":"1900eb00-365d-401d-a399-946fb0c03bfa","Type":"ContainerStarted","Data":"eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1"}
Nov 02 08:17:27.607267 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:27.607233    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" event={"ID":"ed94547d-f04f-4401-a356-fd38ffdcf2e4","Type":"ContainerStarted","Data":"19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d"}
Nov 02 08:17:28.075580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:28.075543    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:17:28.076086 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:28.076002    4898 patch_prober.go:29] interesting pod/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Nov 02 08:17:28.076291 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:28.076266    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Nov 02 08:17:28.525178 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:28.525143    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:17:28.525401 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:28.525249    4898 patch_prober.go:29] interesting pod/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 08:17:28.525401 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:28.525375    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 08:17:29.070338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:29.070300    4898 patch_prober.go:29] interesting pod/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Nov 02 08:17:29.070993 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:29.070950    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Nov 02 08:17:29.540913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:29.532427    4898 patch_prober.go:29] interesting pod/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 08:17:29.540913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:29.532527    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 08:17:30.070475 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:30.070437    4898 patch_prober.go:29] interesting pod/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Nov 02 08:17:30.071151 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:30.071102    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Nov 02 08:17:30.525702 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:30.525664    4898 patch_prober.go:29] interesting pod/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 08:17:30.526068 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:30.525736    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 08:17:31.068345 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:31.068302    4898 patch_prober.go:29] interesting pod/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Nov 02 08:17:31.068681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:31.068654    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Nov 02 08:17:31.524200 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:31.524165    4898 patch_prober.go:29] interesting pod/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 08:17:31.524853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:31.524806    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 08:17:32.071915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:32.071344    4898 patch_prober.go:29] interesting pod/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Nov 02 08:17:32.071915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:32.071419    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Nov 02 08:17:32.528912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:32.528827    4898 patch_prober.go:29] interesting pod/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 08:17:32.529449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:32.528936    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 08:17:33.068315 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:33.068274    4898 patch_prober.go:29] interesting pod/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused" start-of-body=
Nov 02 08:17:33.068633 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:33.068599    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.97:15021/healthz/ready\": dial tcp 10.132.1.97:15021: connect: connection refused"
Nov 02 08:17:33.530464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:33.530429    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:17:33.531028 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:33.531008    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:17:33.531760 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:33.531733    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:17:34.073747 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:34.073712    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:17:34.074085 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:34.074065    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:17:34.074723 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:34.074702    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:17:45.031527 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:17:45.031486    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:17:51.004176 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.004122    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"]
Nov 02 08:17:51.010645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.010609    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.029002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.028963    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"]
Nov 02 08:17:51.042536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.038749    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ba44ed-openshift-default-dockercfg-f6f29\""
Nov 02 08:17:51.042536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.039576    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkfnq\" (UniqueName: \"kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-kube-api-access-gkfnq\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.042536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.039622    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-credential-socket\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.042536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.039797    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-certs\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.042536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.039856    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-podinfo\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.042536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.039982    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-envoy\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.042536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.040100    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-socket\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.042536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.040186    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-token\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.042536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.040337    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcce1599-5df4-4e0a-853a-6127ac605ffe-istiod-ca-cert\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.042536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.040379    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-data\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.141392 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.141352    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-socket\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.141595 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.141415    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-token\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.141595 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.141462    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcce1599-5df4-4e0a-853a-6127ac605ffe-istiod-ca-cert\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.141595 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.141488    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-data\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.141836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.141656    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkfnq\" (UniqueName: \"kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-kube-api-access-gkfnq\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.141836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.141683    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-credential-socket\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.141836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.141753    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-certs\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.141836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.141791    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-podinfo\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.142131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.141835    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-envoy\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.142131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.141992    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-socket\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.142229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.142130    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-credential-socket\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.142229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.142186    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-certs\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.142742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.142715    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-data\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.142964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.142853    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcce1599-5df4-4e0a-853a-6127ac605ffe-istiod-ca-cert\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.144630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.144604    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-envoy\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.144747 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.144617    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-podinfo\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.170090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.170048    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkfnq\" (UniqueName: \"kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-kube-api-access-gkfnq\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.171293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.171263    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-token\") pod \"e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") " pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.333960 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.331171    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:51.868688 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:17:51.868658    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcce1599_5df4_4e0a_853a_6127ac605ffe.slice/crio-2605480c11c97fdd3b899ff091c422592935a951e666a6753fe7a621f486c750 WatchSource:0}: Error finding container 2605480c11c97fdd3b899ff091c422592935a951e666a6753fe7a621f486c750: Status 404 returned error can't find the container with id 2605480c11c97fdd3b899ff091c422592935a951e666a6753fe7a621f486c750
Nov 02 08:17:51.869663 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.869636    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"]
Nov 02 08:17:51.872178 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.872130    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:17:51.872295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.872222    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:17:51.872295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:51.872273    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:17:52.108196 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:52.108153    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" event={"ID":"dcce1599-5df4-4e0a-853a-6127ac605ffe","Type":"ContainerStarted","Data":"4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b"}
Nov 02 08:17:52.108737 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:52.108704    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" event={"ID":"dcce1599-5df4-4e0a-853a-6127ac605ffe","Type":"ContainerStarted","Data":"2605480c11c97fdd3b899ff091c422592935a951e666a6753fe7a621f486c750"}
Nov 02 08:17:52.182286 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:52.176941    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" podStartSLOduration=2.176917349 podStartE2EDuration="2.176917349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:52.152875202 +0000 UTC m=+3222.843202249" watchObservedRunningTime="2026-11-02 08:17:52.176917349 +0000 UTC m=+3222.867244313"
Nov 02 08:17:52.335915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:52.333090    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:52.335915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:52.335480    4898 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body=
Nov 02 08:17:52.335915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:52.335535    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused"
Nov 02 08:17:53.337185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:53.337150    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:54.032681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:54.032644    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"]
Nov 02 08:17:54.044969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:54.044936    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ca4ac6-x9g2g"]
Nov 02 08:17:54.130171 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:54.130137    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:54.131258 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:54.131233    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:17:55.471175 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:55.471136    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e373491d-f04b-4457-9b15-00131cf2b58e" path="/var/lib/kubelet/pods/e373491d-f04b-4457-9b15-00131cf2b58e/volumes"
Nov 02 08:17:56.525623 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.525579    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"]
Nov 02 08:17:56.536907 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.535174    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"
Nov 02 08:17:56.560963 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.559202    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ba44ed\""
Nov 02 08:17:56.560963 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.559202    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ba44ed-dockercfg-qmj8m\""
Nov 02 08:17:56.560963 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.560026    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"]
Nov 02 08:17:56.672919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.662609    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c640c1dc-baae-4bb0-acdb-a5b09e732e55-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z\" (UID: \"c640c1dc-baae-4bb0-acdb-a5b09e732e55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"
Nov 02 08:17:56.672919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.668723    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpvn\" (UniqueName: \"kubernetes.io/projected/c640c1dc-baae-4bb0-acdb-a5b09e732e55-kube-api-access-dtpvn\") pod \"payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z\" (UID: \"c640c1dc-baae-4bb0-acdb-a5b09e732e55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"
Nov 02 08:17:56.802001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.798431    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c640c1dc-baae-4bb0-acdb-a5b09e732e55-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z\" (UID: \"c640c1dc-baae-4bb0-acdb-a5b09e732e55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"
Nov 02 08:17:56.802001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.798554    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpvn\" (UniqueName: \"kubernetes.io/projected/c640c1dc-baae-4bb0-acdb-a5b09e732e55-kube-api-access-dtpvn\") pod \"payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z\" (UID: \"c640c1dc-baae-4bb0-acdb-a5b09e732e55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"
Nov 02 08:17:56.802001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.800114    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c640c1dc-baae-4bb0-acdb-a5b09e732e55-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z\" (UID: \"c640c1dc-baae-4bb0-acdb-a5b09e732e55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"
Nov 02 08:17:56.904920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.902932    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"]
Nov 02 08:17:56.946910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.942590    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"
Nov 02 08:17:57.007382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:56.999669    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpvn\" (UniqueName: \"kubernetes.io/projected/c640c1dc-baae-4bb0-acdb-a5b09e732e55-kube-api-access-dtpvn\") pod \"payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z\" (UID: \"c640c1dc-baae-4bb0-acdb-a5b09e732e55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"
Nov 02 08:17:57.032973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:57.009469    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nrjz\" (UniqueName: \"kubernetes.io/projected/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-kube-api-access-8nrjz\") pod \"payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht\" (UID: \"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"
Nov 02 08:17:57.069344 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:57.069261    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht\" (UID: \"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"
Nov 02 08:17:57.088473 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:57.082369    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"]
Nov 02 08:17:57.161596 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:57.161559    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"
Nov 02 08:17:57.177469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:57.177428    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nrjz\" (UniqueName: \"kubernetes.io/projected/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-kube-api-access-8nrjz\") pod \"payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht\" (UID: \"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"
Nov 02 08:17:57.177675 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:57.177649    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht\" (UID: \"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"
Nov 02 08:17:57.178538 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:57.178517    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht\" (UID: \"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"
Nov 02 08:17:57.201212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:57.201180    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nrjz\" (UniqueName: \"kubernetes.io/projected/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-kube-api-access-8nrjz\") pod \"payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht\" (UID: \"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"
Nov 02 08:17:57.384272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:57.384172    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"
Nov 02 08:17:57.477568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:57.477535    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"]
Nov 02 08:17:57.704295 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:17:57.704260    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2428c6e0_6a9e_4bf3_bc9d_9ec82d8afab8.slice/crio-8a8ded5c668aac309699554974e629986c47e383997043af07c18db2ef24d211 WatchSource:0}: Error finding container 8a8ded5c668aac309699554974e629986c47e383997043af07c18db2ef24d211: Status 404 returned error can't find the container with id 8a8ded5c668aac309699554974e629986c47e383997043af07c18db2ef24d211
Nov 02 08:17:57.707062 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:57.707039    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"]
Nov 02 08:17:58.171210 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:58.171089    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z" event={"ID":"c640c1dc-baae-4bb0-acdb-a5b09e732e55","Type":"ContainerStarted","Data":"1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882"}
Nov 02 08:17:58.171210 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:58.171154    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z" event={"ID":"c640c1dc-baae-4bb0-acdb-a5b09e732e55","Type":"ContainerStarted","Data":"a49a5fcc1652fe9936fd62a5610d38540a59446a89b09cbe8090dba542941011"}
Nov 02 08:17:58.171546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:58.171186    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"
Nov 02 08:17:58.173065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:58.173026    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht" event={"ID":"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8","Type":"ContainerStarted","Data":"a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8"}
Nov 02 08:17:58.173181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:58.173083    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht" event={"ID":"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8","Type":"ContainerStarted","Data":"8a8ded5c668aac309699554974e629986c47e383997043af07c18db2ef24d211"}
Nov 02 08:17:58.173181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:58.173107    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"
Nov 02 08:17:58.224256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:58.224209    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z" podStartSLOduration=2.224196586 podStartE2EDuration="2.224196586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:58.218110926 +0000 UTC m=+3228.908437910" watchObservedRunningTime="2026-11-02 08:17:58.224196586 +0000 UTC m=+3228.914523550"
Nov 02 08:17:58.295490 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:17:58.295438    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht" podStartSLOduration=2.295423165 podStartE2EDuration="2.295423165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:17:58.289357353 +0000 UTC m=+3228.979684320" watchObservedRunningTime="2026-11-02 08:17:58.295423165 +0000 UTC m=+3228.985750168"
Nov 02 08:18:05.032918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:05.032849    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"]
Nov 02 08:18:05.044746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:05.044709    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6468ca-fckkn"]
Nov 02 08:18:05.466976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:05.466937    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c42383-c2e4-4ddc-89dc-c54ca28b5692" path="/var/lib/kubelet/pods/97c42383-c2e4-4ddc-89dc-c54ca28b5692/volumes"
Nov 02 08:18:09.210371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:09.210341    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"
Nov 02 08:18:09.210814 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:09.210426    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"
Nov 02 08:18:14.295346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:14.295307    4898 scope.go:122] "RemoveContainer" containerID="9fd4672ca574d50529f3bbc4582f59b196af9f9012ed582a869389308a1364f7"
Nov 02 08:18:14.308642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:14.308615    4898 scope.go:122] "RemoveContainer" containerID="d65c9ddeaae329195ce23a275b1750e6a1bfbd610285fe1870503d4f24f0339b"
Nov 02 08:18:14.992143 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:18:14.992097    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:18:23.032084 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:23.032047    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"]
Nov 02 08:18:23.044185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:23.044154    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ee127b-7g8r8"]
Nov 02 08:18:23.470511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:23.470473    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033f9df6-5ac0-4d0d-94ed-cc914d7392df" path="/var/lib/kubelet/pods/033f9df6-5ac0-4d0d-94ed-cc914d7392df/volumes"
Nov 02 08:18:36.034650 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:36.034611    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"]
Nov 02 08:18:36.046009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:36.045981    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a970f776-bxtjw"]
Nov 02 08:18:37.029113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:37.029069    4898 patch_prober.go:29] interesting pod/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:18:37.029448 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:37.029413    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:18:37.490463 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:37.490415    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b408bd6-3dbf-4045-8ce5-b0e1e9abd268" path="/var/lib/kubelet/pods/1b408bd6-3dbf-4045-8ce5-b0e1e9abd268/volumes"
Nov 02 08:18:38.948964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:38.948928    4898 generic.go:356] "Generic (PLEG): container finished" podID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerID="681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830" exitCode=0
Nov 02 08:18:38.949488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:38.948994    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" event={"ID":"bf9341cd-5407-4907-a11d-b3af85a6baa5","Type":"ContainerDied","Data":"681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830"}
Nov 02 08:18:38.949641 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:38.949617    4898 scope.go:122] "RemoveContainer" containerID="681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830"
Nov 02 08:18:38.953352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:38.953320    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:18:38.953466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:38.953394    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:18:38.953466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:38.953425    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:18:39.477628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:39.477593    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"]
Nov 02 08:18:39.481523 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:39.481503    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"
Nov 02 08:18:39.499043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:39.499018    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d042a8\""
Nov 02 08:18:39.500144 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:39.499801    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d042a8-dockercfg-rhr2h\""
Nov 02 08:18:39.509738 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:39.509701    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"]
Nov 02 08:18:39.659853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:39.659755    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26618807-5b73-42bf-ad76-b3e5c05a9055-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb\" (UID: \"26618807-5b73-42bf-ad76-b3e5c05a9055\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"
Nov 02 08:18:39.661033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:39.661008    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7j74\" (UniqueName: \"kubernetes.io/projected/26618807-5b73-42bf-ad76-b3e5c05a9055-kube-api-access-v7j74\") pod \"payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb\" (UID: \"26618807-5b73-42bf-ad76-b3e5c05a9055\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"
Nov 02 08:18:39.787193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:39.784701    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26618807-5b73-42bf-ad76-b3e5c05a9055-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb\" (UID: \"26618807-5b73-42bf-ad76-b3e5c05a9055\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"
Nov 02 08:18:39.844068 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:39.804956    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26618807-5b73-42bf-ad76-b3e5c05a9055-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb\" (UID: \"26618807-5b73-42bf-ad76-b3e5c05a9055\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"
Nov 02 08:18:39.844068 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:39.842152    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7j74\" (UniqueName: \"kubernetes.io/projected/26618807-5b73-42bf-ad76-b3e5c05a9055-kube-api-access-v7j74\") pod \"payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb\" (UID: \"26618807-5b73-42bf-ad76-b3e5c05a9055\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"
Nov 02 08:18:39.913993 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:39.913961    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7j74\" (UniqueName: \"kubernetes.io/projected/26618807-5b73-42bf-ad76-b3e5c05a9055-kube-api-access-v7j74\") pod \"payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb\" (UID: \"26618807-5b73-42bf-ad76-b3e5c05a9055\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"
Nov 02 08:18:39.969622 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:39.969585    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" event={"ID":"bf9341cd-5407-4907-a11d-b3af85a6baa5","Type":"ContainerStarted","Data":"ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec"}
Nov 02 08:18:40.099938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:40.099826    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"
Nov 02 08:18:40.447287 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:18:40.447255    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26618807_5b73_42bf_ad76_b3e5c05a9055.slice/crio-4844d9cffb27539e087dcebe725df2d17c013e3ba35712508b55ae207522e41d WatchSource:0}: Error finding container 4844d9cffb27539e087dcebe725df2d17c013e3ba35712508b55ae207522e41d: Status 404 returned error can't find the container with id 4844d9cffb27539e087dcebe725df2d17c013e3ba35712508b55ae207522e41d
Nov 02 08:18:40.449230 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:40.449203    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"]
Nov 02 08:18:40.942913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:40.940328    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:18:40.951911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:40.949011    4898 patch_prober.go:29] interesting pod/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 02 08:18:40.951911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:40.949074    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 02 08:18:40.993477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:40.993438    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb" event={"ID":"26618807-5b73-42bf-ad76-b3e5c05a9055","Type":"ContainerStarted","Data":"71627ac25cd2e36f41b838aaf7d46fffc9b95293ff744c95cfb520685002673a"}
Nov 02 08:18:40.995276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:40.995246    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb" event={"ID":"26618807-5b73-42bf-ad76-b3e5c05a9055","Type":"ContainerStarted","Data":"4844d9cffb27539e087dcebe725df2d17c013e3ba35712508b55ae207522e41d"}
Nov 02 08:18:40.995470 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:40.995453    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"
Nov 02 08:18:41.043347 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:41.043286    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb" podStartSLOduration=2.043267485 podStartE2EDuration="2.043267485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:41.038531203 +0000 UTC m=+3271.728858193" watchObservedRunningTime="2026-11-02 08:18:41.043267485 +0000 UTC m=+3271.733594450"
Nov 02 08:18:41.939582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:41.939542    4898 patch_prober.go:29] interesting pod/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 02 08:18:41.939769 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:41.939602    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 02 08:18:42.939524 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:42.939487    4898 patch_prober.go:29] interesting pod/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 02 08:18:42.939977 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:42.939545    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 02 08:18:43.939111 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:43.939076    4898 patch_prober.go:29] interesting pod/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 02 08:18:43.939377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:43.939127    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 02 08:18:44.939939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:44.939875    4898 patch_prober.go:29] interesting pod/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 02 08:18:44.940354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:44.939956    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 02 08:18:45.008201 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:18:45.008160    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:18:45.944904 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:45.944868    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:18:45.945429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:45.945409    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:18:45.946169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:45.946148    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:18:50.939366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:50.936026    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"]
Nov 02 08:18:50.948996 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:50.948348    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:50.982833 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:50.982796    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"]
Nov 02 08:18:50.991691 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:50.989947    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-eb6010-openshift-default-dockercfg-hq78l\""
Nov 02 08:18:51.141911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.140906    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-credential-socket\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.141911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.141022    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-data\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.141911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.141095    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-envoy\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.141911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.141124    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-socket\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.141911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.141169    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-podinfo\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.141911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.141301    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-certs\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.141911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.141362    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-token\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.141911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.141446    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9474d0bd-4733-44e8-a308-e15f2edb4108-istiod-ca-cert\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.141911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.141473    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwqfp\" (UniqueName: \"kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-kube-api-access-pwqfp\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.243019 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.242847    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9474d0bd-4733-44e8-a308-e15f2edb4108-istiod-ca-cert\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.243278 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.243251    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwqfp\" (UniqueName: \"kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-kube-api-access-pwqfp\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.259911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.243524    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-credential-socket\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.260186 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.260162    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-data\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.260368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.260352    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-envoy\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.260474 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.260462    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-socket\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.260592 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.260578    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-podinfo\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.260815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.260802    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-certs\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.260936 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.252332    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-credential-socket\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.261006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.245735    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9474d0bd-4733-44e8-a308-e15f2edb4108-istiod-ca-cert\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.261519 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.261497    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-certs\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.273974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.265123    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-token\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.273974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.265714    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-data\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.273974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.265983    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-socket\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.300932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.300431    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-podinfo\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.329913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.318584    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-envoy\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.329913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.319918    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-token\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.356918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.334632    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwqfp\" (UniqueName: \"kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-kube-api-access-pwqfp\") pod \"e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") " pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.599485 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.599384    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:51.927475 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:18:51.927444    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9474d0bd_4733_44e8_a308_e15f2edb4108.slice/crio-c57106053f7e378c93d8088a3c7b1d15004d1ca8e4ef1b420b156fea3c240fb8 WatchSource:0}: Error finding container c57106053f7e378c93d8088a3c7b1d15004d1ca8e4ef1b420b156fea3c240fb8: Status 404 returned error can't find the container with id c57106053f7e378c93d8088a3c7b1d15004d1ca8e4ef1b420b156fea3c240fb8
Nov 02 08:18:51.928796 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.928771    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"]
Nov 02 08:18:51.931009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.930973    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:18:51.931122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.931044    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:18:51.931122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:51.931074    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:18:52.010695 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:52.010668    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"
Nov 02 08:18:52.303108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:52.303027    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" event={"ID":"9474d0bd-4733-44e8-a308-e15f2edb4108","Type":"ContainerStarted","Data":"58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a"}
Nov 02 08:18:52.303108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:52.303089    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" event={"ID":"9474d0bd-4733-44e8-a308-e15f2edb4108","Type":"ContainerStarted","Data":"c57106053f7e378c93d8088a3c7b1d15004d1ca8e4ef1b420b156fea3c240fb8"}
Nov 02 08:18:52.373535 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:52.360043    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" podStartSLOduration=2.360024347 podStartE2EDuration="2.360024347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:18:52.355179651 +0000 UTC m=+3283.045506639" watchObservedRunningTime="2026-11-02 08:18:52.360024347 +0000 UTC m=+3283.050351341"
Nov 02 08:18:52.599562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:52.599473    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:53.606411 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:53.606383    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:54.364837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:54.364803    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:18:54.365684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:18:54.365659    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:19:10.592407 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:10.592373    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:19:10.598471 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:10.598440    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:19:14.369600 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:14.369567    4898 scope.go:122] "RemoveContainer" containerID="269af8961e02fced5a07a31439b82c2da88fc8383ed90ee4d0e6ab56e96cbfbd"
Nov 02 08:19:14.397546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:14.397524    4898 scope.go:122] "RemoveContainer" containerID="21658e22d41edc0acc6ae1f1de8f1437fa20ff4dab728a2c744b47437a47e649"
Nov 02 08:19:15.009092 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:19:15.009050    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:19:21.712490 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:21.712455    4898 generic.go:356] "Generic (PLEG): container finished" podID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerID="4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b" exitCode=0
Nov 02 08:19:21.713005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:21.712526    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" event={"ID":"dcce1599-5df4-4e0a-853a-6127ac605ffe","Type":"ContainerDied","Data":"4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b"}
Nov 02 08:19:21.713005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:21.713002    4898 scope.go:122] "RemoveContainer" containerID="4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b"
Nov 02 08:19:21.715138 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:21.715105    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:19:21.715253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:21.715184    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:19:21.715253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:21.715214    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:19:22.724505 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:22.724466    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" event={"ID":"dcce1599-5df4-4e0a-853a-6127ac605ffe","Type":"ContainerStarted","Data":"61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64"}
Nov 02 08:19:23.335344 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:23.335307    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:19:23.349062 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:23.349014    4898 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body=
Nov 02 08:19:23.349478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:23.349444    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused"
Nov 02 08:19:24.130626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:24.130582    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:19:24.332353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:24.332313    4898 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body=
Nov 02 08:19:24.332575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:24.332383    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused"
Nov 02 08:19:25.245069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:25.245023    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"]
Nov 02 08:19:25.246297 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:25.246251    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" podUID="717e0316-0675-4f60-ba3e-e51e854cfc72" containerName="istio-proxy" containerID="cri-o://e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602" gracePeriod=30
Nov 02 08:19:25.332436 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:25.332402    4898 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body=
Nov 02 08:19:25.332611 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:25.332453    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused"
Nov 02 08:19:26.331800 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:26.331761    4898 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body=
Nov 02 08:19:26.332231 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:26.331818    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused"
Nov 02 08:19:27.332023 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:27.331986    4898 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body=
Nov 02 08:19:27.332469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:27.332046    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused"
Nov 02 08:19:28.337277 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:28.337242    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:19:28.338324 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:28.338297    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:19:30.640964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.640921    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:19:30.744875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.744839    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-envoy\") pod \"717e0316-0675-4f60-ba3e-e51e854cfc72\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") "
Nov 02 08:19:30.745134 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.744960    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-certs\") pod \"717e0316-0675-4f60-ba3e-e51e854cfc72\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") "
Nov 02 08:19:30.745134 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.745000    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-token\" (UniqueName: \"kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-token\") pod \"717e0316-0675-4f60-ba3e-e51e854cfc72\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") "
Nov 02 08:19:30.745134 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.745075    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717e0316-0675-4f60-ba3e-e51e854cfc72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717e0316-0675-4f60-ba3e-e51e854cfc72-istiod-ca-cert\") pod \"717e0316-0675-4f60-ba3e-e51e854cfc72\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") "
Nov 02 08:19:30.745134 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.745111    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-credential-socket\") pod \"717e0316-0675-4f60-ba3e-e51e854cfc72\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") "
Nov 02 08:19:30.745367 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.745136    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-data\") pod \"717e0316-0675-4f60-ba3e-e51e854cfc72\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") "
Nov 02 08:19:30.745367 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.745167    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-socket\") pod \"717e0316-0675-4f60-ba3e-e51e854cfc72\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") "
Nov 02 08:19:30.745750 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.745716    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-certs" pod "717e0316-0675-4f60-ba3e-e51e854cfc72" (UID: "717e0316-0675-4f60-ba3e-e51e854cfc72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:19:30.745837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.745766    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717e0316-0675-4f60-ba3e-e51e854cfc72-istiod-ca-cert" pod "717e0316-0675-4f60-ba3e-e51e854cfc72" (UID: "717e0316-0675-4f60-ba3e-e51e854cfc72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:19:30.745837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.745823    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-podinfo\") pod \"717e0316-0675-4f60-ba3e-e51e854cfc72\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") "
Nov 02 08:19:30.746020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.746000    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-kube-api-access-24vc9\" (UniqueName: \"kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-kube-api-access-24vc9\") pod \"717e0316-0675-4f60-ba3e-e51e854cfc72\" (UID: \"717e0316-0675-4f60-ba3e-e51e854cfc72\") "
Nov 02 08:19:30.746603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.746573    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-socket" pod "717e0316-0675-4f60-ba3e-e51e854cfc72" (UID: "717e0316-0675-4f60-ba3e-e51e854cfc72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:19:30.746850 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.746825    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-data" pod "717e0316-0675-4f60-ba3e-e51e854cfc72" (UID: "717e0316-0675-4f60-ba3e-e51e854cfc72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:19:30.746977 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.746956    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-credential-socket" pod "717e0316-0675-4f60-ba3e-e51e854cfc72" (UID: "717e0316-0675-4f60-ba3e-e51e854cfc72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:19:30.747230 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.747209    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:19:30.747316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.747236    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717e0316-0675-4f60-ba3e-e51e854cfc72-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:19:30.747316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.747250    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:19:30.747316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.747264    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:19:30.747316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.747277    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:19:30.749217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.749184    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-kube-api-access-24vc9" pod "717e0316-0675-4f60-ba3e-e51e854cfc72" (UID: "717e0316-0675-4f60-ba3e-e51e854cfc72"). InnerVolumeSpecName "kube-api-access-24vc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:19:30.753429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.753374    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-token" pod "717e0316-0675-4f60-ba3e-e51e854cfc72" (UID: "717e0316-0675-4f60-ba3e-e51e854cfc72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:19:30.754214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.753881    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-podinfo" pod "717e0316-0675-4f60-ba3e-e51e854cfc72" (UID: "717e0316-0675-4f60-ba3e-e51e854cfc72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:19:30.754311 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.754295    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-envoy" pod "717e0316-0675-4f60-ba3e-e51e854cfc72" (UID: "717e0316-0675-4f60-ba3e-e51e854cfc72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:19:30.832740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.832655    4898 generic.go:356] "Generic (PLEG): container finished" podID="717e0316-0675-4f60-ba3e-e51e854cfc72" containerID="e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602" exitCode=0
Nov 02 08:19:30.832740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.832706    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"
Nov 02 08:19:30.832982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.832739    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" event={"ID":"717e0316-0675-4f60-ba3e-e51e854cfc72","Type":"ContainerDied","Data":"e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602"}
Nov 02 08:19:30.832982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.832798    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9" event={"ID":"717e0316-0675-4f60-ba3e-e51e854cfc72","Type":"ContainerDied","Data":"6052f4ca9f4e6016821cd78c694ff43dc8555223995d404a9b9c55902e83dba6"}
Nov 02 08:19:30.832982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.832822    4898 scope.go:122] "RemoveContainer" containerID="e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602"
Nov 02 08:19:30.848721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.848680    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:19:30.848721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.848715    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24vc9\" (UniqueName: \"kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-kube-api-access-24vc9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:19:30.848939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.848730    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:19:30.848939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.848746    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717e0316-0675-4f60-ba3e-e51e854cfc72-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:19:30.849543 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.849524    4898 scope.go:122] "RemoveContainer" containerID="e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602"
Nov 02 08:19:30.849914 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:19:30.849878    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602\": container with ID starting with e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602 not found: ID does not exist" containerID="e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602"
Nov 02 08:19:30.849992 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.849921    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602"} err="failed to get container status \"e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602\": rpc error: code = NotFound desc = could not find container \"e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602\": container with ID starting with e9f7fff24444849839091761d32459dc4ed5dec863f1cc2328f6513fc27e2602 not found: ID does not exist"
Nov 02 08:19:30.898325 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.898279    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"]
Nov 02 08:19:30.903702 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:30.903665    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jvnf9"]
Nov 02 08:19:31.469192 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:19:31.469145    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717e0316-0675-4f60-ba3e-e51e854cfc72" path="/var/lib/kubelet/pods/717e0316-0675-4f60-ba3e-e51e854cfc72/volumes"
Nov 02 08:19:44.992608 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:19:44.992566    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:20:14.454950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:14.454885    4898 scope.go:122] "RemoveContainer" containerID="0ef87bfc62b0d4b09398537fa0717c077a792d9484892ebcd04814932107223d"
Nov 02 08:20:14.469442 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:14.469413    4898 scope.go:122] "RemoveContainer" containerID="b951da923ae84088d01b9eff0f351ed1a1bb7e7b1066ac9f34dd51ac39c0e7ab"
Nov 02 08:20:14.992449 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:20:14.992400    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:20:26.789171 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.789062    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"]
Nov 02 08:20:26.790435 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.790406    4898 cpu_manager.go:414] "containerMap: removing container" podUID="717e0316-0675-4f60-ba3e-e51e854cfc72" containerName="istio-proxy"
Nov 02 08:20:26.790552 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.790437    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717e0316-0675-4f60-ba3e-e51e854cfc72" containerName="istio-proxy"
Nov 02 08:20:26.790699 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.790683    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="717e0316-0675-4f60-ba3e-e51e854cfc72" containerName="istio-proxy"
Nov 02 08:20:26.805949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.805856    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"]
Nov 02 08:20:26.806116 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.806023    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:26.821756 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.821690    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ba44ed\"/\"kube-root-ca.crt\""
Nov 02 08:20:26.834046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.834019    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ba44ed\"/\"openshift-service-ca.crt\""
Nov 02 08:20:26.834493 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.834337    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ba44ed\"/\"test-model-ba44ed-kserve-self-signed-certs\""
Nov 02 08:20:26.834656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.834436    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ba44ed\"/\"default-dockercfg-n9p6b\""
Nov 02 08:20:26.919090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.919048    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-dshm\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:26.919286 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.919213    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbgvv\" (UniqueName: \"kubernetes.io/projected/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-kube-api-access-pbgvv\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:26.919286 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.919279    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-model-cache\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:26.919381 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.919359    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tls-certs\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:26.919559 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.919541    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tmp-dir\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:26.919679 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:26.919650    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-home\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.072977 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.072397    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbgvv\" (UniqueName: \"kubernetes.io/projected/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-kube-api-access-pbgvv\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.072977 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.072494    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-model-cache\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.072977 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.072557    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tls-certs\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.072977 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.072741    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tmp-dir\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.072977 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.072817    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-home\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.072977 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.072965    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-dshm\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.125913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.097131    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"]
Nov 02 08:20:27.125913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.102338    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tmp-dir\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.125913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.103348    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-home\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.125913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.103607    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-model-cache\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.125913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.109571    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-dshm\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.151946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.138651    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbgvv\" (UniqueName: \"kubernetes.io/projected/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-kube-api-access-pbgvv\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.186428 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.164871    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tls-certs\") pod \"test-model-ba44ed-kserve-659c6df84c-spdfp\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") " pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.186428 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.170992    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfc62b0a-kwhk6"]
Nov 02 08:20:27.438712 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.438679    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:27.504912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.498057    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f504d5-53f9-40ec-bf39-074da613f90b" path="/var/lib/kubelet/pods/74f504d5-53f9-40ec-bf39-074da613f90b/volumes"
Nov 02 08:20:27.888679 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:20:27.888647    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc741a4f_4c12_41e4_98e7_6a9231c6f00e.slice/crio-78aefa919b67993e112b93e75e42a199ba952d618de831c325603c845f9aac3f WatchSource:0}: Error finding container 78aefa919b67993e112b93e75e42a199ba952d618de831c325603c845f9aac3f: Status 404 returned error can't find the container with id 78aefa919b67993e112b93e75e42a199ba952d618de831c325603c845f9aac3f
Nov 02 08:20:27.891775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:27.891745    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"]
Nov 02 08:20:28.647353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:28.647310    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp" event={"ID":"cc741a4f-4c12-41e4-98e7-6a9231c6f00e","Type":"ContainerStarted","Data":"43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b"}
Nov 02 08:20:28.647543 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:28.647365    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp" event={"ID":"cc741a4f-4c12-41e4-98e7-6a9231c6f00e","Type":"ContainerStarted","Data":"78aefa919b67993e112b93e75e42a199ba952d618de831c325603c845f9aac3f"}
Nov 02 08:20:28.696965 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:28.696881    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp" podStartSLOduration=2.696860525 podStartE2EDuration="2.696860525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:20:28.690081191 +0000 UTC m=+3379.380408154" watchObservedRunningTime="2026-11-02 08:20:28.696860525 +0000 UTC m=+3379.387187490"
Nov 02 08:20:29.439237 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:29.439198    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:35.745557 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:35.745521    4898 generic.go:356] "Generic (PLEG): container finished" podID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerID="58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a" exitCode=0
Nov 02 08:20:35.746055 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:35.745593    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" event={"ID":"9474d0bd-4733-44e8-a308-e15f2edb4108","Type":"ContainerDied","Data":"58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a"}
Nov 02 08:20:35.746055 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:35.746027    4898 scope.go:122] "RemoveContainer" containerID="58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a"
Nov 02 08:20:35.748160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:35.748112    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:20:35.748270 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:35.748198    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:20:35.748270 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:35.748232    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:20:36.768955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:36.756700    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" event={"ID":"9474d0bd-4733-44e8-a308-e15f2edb4108","Type":"ContainerStarted","Data":"b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f"}
Nov 02 08:20:37.439988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:37.439954    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:37.498474 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:37.498442    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:37.599847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:37.599810    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:20:37.601003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:37.600970    4898 patch_prober.go:29] interesting pod/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 08:20:37.601146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:37.601041    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 08:20:37.811504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:37.811418    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:20:38.033592 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:38.033552    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"]
Nov 02 08:20:38.038365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:38.038335    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c93a6d-8bn9c"]
Nov 02 08:20:38.601290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:38.601249    4898 patch_prober.go:29] interesting pod/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 08:20:38.601478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:38.601318    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 08:20:39.373915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:39.372981    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:20:39.496277 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:39.496196    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e20572-c6b6-4a4f-9519-873d9b7a2b7a" path="/var/lib/kubelet/pods/17e20572-c6b6-4a4f-9519-873d9b7a2b7a/volumes"
Nov 02 08:20:39.599901 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:39.599850    4898 patch_prober.go:29] interesting pod/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 08:20:39.600108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:39.599961    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 08:20:40.599702 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:40.599665    4898 patch_prober.go:29] interesting pod/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 08:20:40.600228 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:40.599738    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 08:20:41.600489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:41.600449    4898 patch_prober.go:29] interesting pod/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 08:20:41.601086 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:41.600517    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 08:20:42.691669 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:42.690973    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:20:42.693806 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:42.693778    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:20:44.991004 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:20:44.990961    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:20:59.051929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:59.050675    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"]
Nov 02 08:20:59.058720 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:59.058664    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0703208e-mdd8b"]
Nov 02 08:20:59.468668 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:20:59.468629    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="513e3508-7224-4fc1-ad0c-13db184d0934" path="/var/lib/kubelet/pods/513e3508-7224-4fc1-ad0c-13db184d0934/volumes"
Nov 02 08:21:01.034056 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:01.034023    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"]
Nov 02 08:21:01.045513 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:01.045486    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2510c0c-8k6lj"]
Nov 02 08:21:01.469426 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:01.469393    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa104e2-5087-434a-ac18-459608d49e6f" path="/var/lib/kubelet/pods/8fa104e2-5087-434a-ac18-459608d49e6f/volumes"
Nov 02 08:21:09.031929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:09.031857    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"]
Nov 02 08:21:09.043449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:09.043417    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dac6077-5kvv4"]
Nov 02 08:21:09.471640 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:09.471597    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dbbb25f-7059-46bc-bdaf-047b816dfd5b" path="/var/lib/kubelet/pods/9dbbb25f-7059-46bc-bdaf-047b816dfd5b/volumes"
Nov 02 08:21:12.032904 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:12.032849    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"]
Nov 02 08:21:12.044240 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:12.044210    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4bfdd88-42bpc"]
Nov 02 08:21:13.479809 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:13.479770    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="438f69a6-2769-494d-b91d-3ba742d57b0d" path="/var/lib/kubelet/pods/438f69a6-2769-494d-b91d-3ba742d57b0d/volumes"
Nov 02 08:21:14.994813 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:21:14.994768    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:21:23.034808 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:23.034768    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"]
Nov 02 08:21:23.046434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:23.046408    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d35571-6hkkl"]
Nov 02 08:21:23.469506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:23.469462    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68463862-7d8c-468c-b22a-14783ed38fe3" path="/var/lib/kubelet/pods/68463862-7d8c-468c-b22a-14783ed38fe3/volumes"
Nov 02 08:21:34.032744 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.032705    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"]
Nov 02 08:21:34.044262 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.044228    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf093132-7gr5m"]
Nov 02 08:21:34.123234 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.123201    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"]
Nov 02 08:21:34.129263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.129243    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.215261 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.215225    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"]
Nov 02 08:21:34.232008 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.231979    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d042a8-openshift-default-dockercfg-dgvtx\""
Nov 02 08:21:34.266212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.266172    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-socket\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.266410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.266239    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-credential-socket\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.266410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.266281    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-podinfo\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.266410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.266359    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-certs\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.266410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.266393    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istiod-ca-cert\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.266645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.266418    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4frj\" (UniqueName: \"kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-kube-api-access-d4frj\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.266645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.266509    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-token\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.266766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.266671    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-envoy\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.266766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.266724    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-data\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.368056 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.367968    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-data\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.368253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.368078    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-socket\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.368253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.368111    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-credential-socket\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.368253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.368134    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-podinfo\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.368253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.368203    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-certs\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.368253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.368228    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istiod-ca-cert\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.368253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.368252    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4frj\" (UniqueName: \"kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-kube-api-access-d4frj\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.368587 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.368380    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-token\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.368587 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.368436    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-data\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.368587 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.368474    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-socket\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.368587 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.368516    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-credential-socket\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.368810 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.368681    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-envoy\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.369100 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.369074    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-certs\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.369203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.369142    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istiod-ca-cert\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.370833 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.370809    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-envoy\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.371344 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.371326    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-podinfo\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.392500 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.392476    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-token\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.392786 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.392761    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4frj\" (UniqueName: \"kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-kube-api-access-d4frj\") pod \"e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") " pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.444557 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.444529    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:34.780097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.780069    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"]
Nov 02 08:21:34.783476 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:21:34.783444    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1fd0fb_6cda_4a52_afcf_2d0698baf983.slice/crio-5c040e1c8615feb6cea672816d63e364fc9546d1c2a1e998ee86f113e2a38831 WatchSource:0}: Error finding container 5c040e1c8615feb6cea672816d63e364fc9546d1c2a1e998ee86f113e2a38831: Status 404 returned error can't find the container with id 5c040e1c8615feb6cea672816d63e364fc9546d1c2a1e998ee86f113e2a38831
Nov 02 08:21:34.787080 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.787041    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:21:34.787195 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.787113    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:21:34.787195 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:34.787148    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:21:35.517912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:35.517742    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3684e5-9c55-4ae6-9b49-6e48ade8686e" path="/var/lib/kubelet/pods/6a3684e5-9c55-4ae6-9b49-6e48ade8686e/volumes"
Nov 02 08:21:35.569630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:35.569584    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr" event={"ID":"7e1fd0fb-6cda-4a52-afcf-2d0698baf983","Type":"ContainerStarted","Data":"20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9"}
Nov 02 08:21:35.569829 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:35.569658    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr" event={"ID":"7e1fd0fb-6cda-4a52-afcf-2d0698baf983","Type":"ContainerStarted","Data":"5c040e1c8615feb6cea672816d63e364fc9546d1c2a1e998ee86f113e2a38831"}
Nov 02 08:21:35.631986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:35.631934    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr" podStartSLOduration=1.631919391 podStartE2EDuration="1.631919391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:21:35.62671755 +0000 UTC m=+3446.317044542" watchObservedRunningTime="2026-11-02 08:21:35.631919391 +0000 UTC m=+3446.322246352"
Nov 02 08:21:36.444921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:36.444847    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:36.446946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:36.446916    4898 patch_prober.go:29] interesting pod/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body=
Nov 02 08:21:36.447092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:36.446965    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr" podUID="7e1fd0fb-6cda-4a52-afcf-2d0698baf983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused"
Nov 02 08:21:37.445293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:37.445257    4898 patch_prober.go:29] interesting pod/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body=
Nov 02 08:21:37.445781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:37.445323    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr" podUID="7e1fd0fb-6cda-4a52-afcf-2d0698baf983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused"
Nov 02 08:21:38.445716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:38.445646    4898 patch_prober.go:29] interesting pod/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body=
Nov 02 08:21:38.446137 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:38.445737    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr" podUID="7e1fd0fb-6cda-4a52-afcf-2d0698baf983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused"
Nov 02 08:21:39.445686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:39.445647    4898 patch_prober.go:29] interesting pod/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body=
Nov 02 08:21:39.445876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:39.445727    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr" podUID="7e1fd0fb-6cda-4a52-afcf-2d0698baf983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused"
Nov 02 08:21:40.445050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:40.445008    4898 patch_prober.go:29] interesting pod/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused" start-of-body=
Nov 02 08:21:40.445220 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:40.445084    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr" podUID="7e1fd0fb-6cda-4a52-afcf-2d0698baf983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.113:15021/healthz/ready\": dial tcp 10.132.1.113:15021: connect: connection refused"
Nov 02 08:21:41.634543 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:41.634505    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:41.635368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:41.635340    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:41.636938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:41.636879    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:21:42.284054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.284017    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"]
Nov 02 08:21:42.289947 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.289928    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:42.308747 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.308718    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"]
Nov 02 08:21:42.313316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.313292    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:21:42.313450 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.313375    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:21:42.313522 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.313485    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:21:42.318161 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.318136    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:21:42.442869 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.442835    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95jn7\" (UniqueName: \"kubernetes.io/projected/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-kube-api-access-95jn7\") pod \"maas-api-revoke-sub-8d906200-dvmpb\" (UID: \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:42.443062 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.442902    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d906200-dvmpb\" (UID: \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:42.443062 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.442982    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d906200-dvmpb\" (UID: \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:42.545103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.545023    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95jn7\" (UniqueName: \"kubernetes.io/projected/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-kube-api-access-95jn7\") pod \"maas-api-revoke-sub-8d906200-dvmpb\" (UID: \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:42.545103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.545084    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d906200-dvmpb\" (UID: \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:42.545285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.545125    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d906200-dvmpb\" (UID: \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:42.545956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.545930    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d906200-dvmpb\" (UID: \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:42.547539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.547514    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d906200-dvmpb\" (UID: \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:42.570615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.570580    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95jn7\" (UniqueName: \"kubernetes.io/projected/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-kube-api-access-95jn7\") pod \"maas-api-revoke-sub-8d906200-dvmpb\" (UID: \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:42.611069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.611041    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:42.938768 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:21:42.938739    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fb71c6_d4e0_408f_bdd1_0c6ce85a9156.slice/crio-c7009a0bfcd1849f2b24098d5de2495d9d1da5a2009c291a200dc3ec221c9e9d WatchSource:0}: Error finding container c7009a0bfcd1849f2b24098d5de2495d9d1da5a2009c291a200dc3ec221c9e9d: Status 404 returned error can't find the container with id c7009a0bfcd1849f2b24098d5de2495d9d1da5a2009c291a200dc3ec221c9e9d
Nov 02 08:21:42.939335 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:42.939311    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"]
Nov 02 08:21:43.680075 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:43.680028    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb" event={"ID":"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156","Type":"ContainerStarted","Data":"3409446a3c879302a2390d8fb8893b0bfbd9278fbf1b0ff34b0f7d9822dee506"}
Nov 02 08:21:43.680274 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:43.680106    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb" event={"ID":"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156","Type":"ContainerStarted","Data":"c7009a0bfcd1849f2b24098d5de2495d9d1da5a2009c291a200dc3ec221c9e9d"}
Nov 02 08:21:43.723139 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:43.723083    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb" podStartSLOduration=1.723069119 podStartE2EDuration="1.723069119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:21:43.719440048 +0000 UTC m=+3454.409767038" watchObservedRunningTime="2026-11-02 08:21:43.723069119 +0000 UTC m=+3454.413396080"
Nov 02 08:21:43.800977 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:43.800942    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"]
Nov 02 08:21:43.801206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:43.801178    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerName="istio-proxy" containerID="cri-o://5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add" gracePeriod=30
Nov 02 08:21:44.692382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:44.692345    4898 generic.go:356] "Generic (PLEG): container finished" podID="36fb71c6-d4e0-408f-bdd1-0c6ce85a9156" containerID="3409446a3c879302a2390d8fb8893b0bfbd9278fbf1b0ff34b0f7d9822dee506" exitCode=0
Nov 02 08:21:44.692839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:44.692424    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb" event={"ID":"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156","Type":"ContainerDied","Data":"3409446a3c879302a2390d8fb8893b0bfbd9278fbf1b0ff34b0f7d9822dee506"}
Nov 02 08:21:44.992808 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:21:44.992708    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:21:46.142278 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.141803    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:46.208714 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.208671    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-xks-ca\") pod \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\" (UID: \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\") "
Nov 02 08:21:46.209221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.209191    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-openshift-ca\") pod \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\" (UID: \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\") "
Nov 02 08:21:46.209448 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.209428    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-kube-api-access-95jn7\" (UniqueName: \"kubernetes.io/projected/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-kube-api-access-95jn7\") pod \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\" (UID: \"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156\") "
Nov 02 08:21:46.241909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.227119    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-openshift-ca" pod "36fb71c6-d4e0-408f-bdd1-0c6ce85a9156" (UID: "36fb71c6-d4e0-408f-bdd1-0c6ce85a9156"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:21:46.241909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.227303    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-xks-ca" pod "36fb71c6-d4e0-408f-bdd1-0c6ce85a9156" (UID: "36fb71c6-d4e0-408f-bdd1-0c6ce85a9156"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:21:46.241909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.228587    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-kube-api-access-95jn7" pod "36fb71c6-d4e0-408f-bdd1-0c6ce85a9156" (UID: "36fb71c6-d4e0-408f-bdd1-0c6ce85a9156"). InnerVolumeSpecName "kube-api-access-95jn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:21:46.312693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.312579    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:46.313037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.312999    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95jn7\" (UniqueName: \"kubernetes.io/projected/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-kube-api-access-95jn7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:46.313186 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.313173    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:46.750705 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.750664    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb" event={"ID":"36fb71c6-d4e0-408f-bdd1-0c6ce85a9156","Type":"ContainerDied","Data":"c7009a0bfcd1849f2b24098d5de2495d9d1da5a2009c291a200dc3ec221c9e9d"}
Nov 02 08:21:46.754914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.750989    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7009a0bfcd1849f2b24098d5de2495d9d1da5a2009c291a200dc3ec221c9e9d"
Nov 02 08:21:46.755182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:46.755161    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"
Nov 02 08:21:49.552854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.552824    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 08:21:49.653213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.653179    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-podinfo\") pod \"4eb2e6c5-145c-4448-8404-1f676175dc7e\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") "
Nov 02 08:21:49.653425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.653234    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-credential-socket\") pod \"4eb2e6c5-145c-4448-8404-1f676175dc7e\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") "
Nov 02 08:21:49.653425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.653363    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-envoy\") pod \"4eb2e6c5-145c-4448-8404-1f676175dc7e\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") "
Nov 02 08:21:49.653541 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.653467    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-kube-api-access-qgtrx\" (UniqueName: \"kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-kube-api-access-qgtrx\") pod \"4eb2e6c5-145c-4448-8404-1f676175dc7e\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") "
Nov 02 08:21:49.653602 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.653576    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-credential-socket" pod "4eb2e6c5-145c-4448-8404-1f676175dc7e" (UID: "4eb2e6c5-145c-4448-8404-1f676175dc7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:21:49.653700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.653682    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-socket\") pod \"4eb2e6c5-145c-4448-8404-1f676175dc7e\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") "
Nov 02 08:21:49.653757 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.653738    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-certs\") pod \"4eb2e6c5-145c-4448-8404-1f676175dc7e\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") "
Nov 02 08:21:49.653813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.653766    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-data\") pod \"4eb2e6c5-145c-4448-8404-1f676175dc7e\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") "
Nov 02 08:21:49.653869 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.653846    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb2e6c5-145c-4448-8404-1f676175dc7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb2e6c5-145c-4448-8404-1f676175dc7e-istiod-ca-cert\") pod \"4eb2e6c5-145c-4448-8404-1f676175dc7e\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") "
Nov 02 08:21:49.654013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.653987    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-socket" pod "4eb2e6c5-145c-4448-8404-1f676175dc7e" (UID: "4eb2e6c5-145c-4448-8404-1f676175dc7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:21:49.654078 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.654050    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-token\" (UniqueName: \"kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-token\") pod \"4eb2e6c5-145c-4448-8404-1f676175dc7e\" (UID: \"4eb2e6c5-145c-4448-8404-1f676175dc7e\") "
Nov 02 08:21:49.654554 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.654488    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-data" pod "4eb2e6c5-145c-4448-8404-1f676175dc7e" (UID: "4eb2e6c5-145c-4448-8404-1f676175dc7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:21:49.654954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.654919    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb2e6c5-145c-4448-8404-1f676175dc7e-istiod-ca-cert" pod "4eb2e6c5-145c-4448-8404-1f676175dc7e" (UID: "4eb2e6c5-145c-4448-8404-1f676175dc7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:21:49.655456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.655374    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:49.655456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.655403    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb2e6c5-145c-4448-8404-1f676175dc7e-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:49.655456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.655417    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:49.655456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.655432    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:49.656101 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.656065    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-certs" pod "4eb2e6c5-145c-4448-8404-1f676175dc7e" (UID: "4eb2e6c5-145c-4448-8404-1f676175dc7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:21:49.656579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.656553    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-kube-api-access-qgtrx" pod "4eb2e6c5-145c-4448-8404-1f676175dc7e" (UID: "4eb2e6c5-145c-4448-8404-1f676175dc7e"). InnerVolumeSpecName "kube-api-access-qgtrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:21:49.656683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.656610    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-token" pod "4eb2e6c5-145c-4448-8404-1f676175dc7e" (UID: "4eb2e6c5-145c-4448-8404-1f676175dc7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:21:49.656739 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.656680    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-podinfo" pod "4eb2e6c5-145c-4448-8404-1f676175dc7e" (UID: "4eb2e6c5-145c-4448-8404-1f676175dc7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:21:49.658266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.658244    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-envoy" pod "4eb2e6c5-145c-4448-8404-1f676175dc7e" (UID: "4eb2e6c5-145c-4448-8404-1f676175dc7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:21:49.757501 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.757460    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:49.757501 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.757505    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:49.757749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.757520    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:49.757749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.757533    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgtrx\" (UniqueName: \"kubernetes.io/projected/4eb2e6c5-145c-4448-8404-1f676175dc7e-kube-api-access-qgtrx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:49.757749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.757549    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb2e6c5-145c-4448-8404-1f676175dc7e-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:49.785299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.785263    4898 generic.go:356] "Generic (PLEG): container finished" podID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerID="5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add" exitCode=0
Nov 02 08:21:49.785480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.785338    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"
Nov 02 08:21:49.785480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.785345    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" event={"ID":"4eb2e6c5-145c-4448-8404-1f676175dc7e","Type":"ContainerDied","Data":"5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add"}
Nov 02 08:21:49.785480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.785464    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5" event={"ID":"4eb2e6c5-145c-4448-8404-1f676175dc7e","Type":"ContainerDied","Data":"5a6be7c30b4d37e463b0dce6042a9c9b9ba7121ac6bc5b9d884d9a9700e322f6"}
Nov 02 08:21:49.785616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.785490    4898 scope.go:122] "RemoveContainer" containerID="5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add"
Nov 02 08:21:49.802200 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.802175    4898 scope.go:122] "RemoveContainer" containerID="5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c"
Nov 02 08:21:49.817499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.817477    4898 scope.go:122] "RemoveContainer" containerID="5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add"
Nov 02 08:21:49.822203 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:21:49.818291    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add\": container with ID starting with 5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add not found: ID does not exist" containerID="5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add"
Nov 02 08:21:49.822203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.818333    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add"} err="failed to get container status \"5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add\": rpc error: code = NotFound desc = could not find container \"5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add\": container with ID starting with 5c708601dc3d5a13a94bdd37f4ee6ef8622514167fe1f389762583f0323b7add not found: ID does not exist"
Nov 02 08:21:49.822203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.818358    4898 scope.go:122] "RemoveContainer" containerID="5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c"
Nov 02 08:21:49.822203 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:21:49.821175    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c\": container with ID starting with 5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c not found: ID does not exist" containerID="5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c"
Nov 02 08:21:49.822203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.821205    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c"} err="failed to get container status \"5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c\": rpc error: code = NotFound desc = could not find container \"5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c\": container with ID starting with 5385447c7e4d5b56d7baf4be86535414f2ed167f06a1c9492385f87ab6e1280c not found: ID does not exist"
Nov 02 08:21:49.845486 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.845452    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"]
Nov 02 08:21:49.850075 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:49.850040    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6929a4-openshift-default-6c5447865-rpnk5"]
Nov 02 08:21:51.474379 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:51.474286    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" path="/var/lib/kubelet/pods/4eb2e6c5-145c-4448-8404-1f676175dc7e/volumes"
Nov 02 08:21:53.757548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.757507    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"]
Nov 02 08:21:53.758570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.758545    4898 cpu_manager.go:414] "containerMap: removing container" podUID="36fb71c6-d4e0-408f-bdd1-0c6ce85a9156" containerName="revoke-keys"
Nov 02 08:21:53.758657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.758573    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36fb71c6-d4e0-408f-bdd1-0c6ce85a9156" containerName="revoke-keys"
Nov 02 08:21:53.758657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.758633    4898 cpu_manager.go:414] "containerMap: removing container" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerName="istio-proxy"
Nov 02 08:21:53.758657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.758641    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerName="istio-proxy"
Nov 02 08:21:53.758874 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.758856    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="36fb71c6-d4e0-408f-bdd1-0c6ce85a9156" containerName="revoke-keys"
Nov 02 08:21:53.758950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.758885    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerName="istio-proxy"
Nov 02 08:21:53.758950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.758910    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerName="istio-proxy"
Nov 02 08:21:53.764368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.764348    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:53.785098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.785066    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"]
Nov 02 08:21:53.789351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.788568    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:21:53.789351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.788681    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:21:53.789351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.788730    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:21:53.789351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.788799    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:21:53.858628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.858583    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5f3e839-5sb6l\" (UID: \"ed378475-ec00-4728-8580-8fc7cd8a57bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:53.858859 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.858831    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5f3e839-5sb6l\" (UID: \"ed378475-ec00-4728-8580-8fc7cd8a57bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:53.859034 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.859015    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chsg8\" (UniqueName: \"kubernetes.io/projected/ed378475-ec00-4728-8580-8fc7cd8a57bc-kube-api-access-chsg8\") pod \"maas-api-revoke-sub-c5f3e839-5sb6l\" (UID: \"ed378475-ec00-4728-8580-8fc7cd8a57bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:53.960640 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.960599    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5f3e839-5sb6l\" (UID: \"ed378475-ec00-4728-8580-8fc7cd8a57bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:53.960847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.960774    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5f3e839-5sb6l\" (UID: \"ed378475-ec00-4728-8580-8fc7cd8a57bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:53.960983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.960874    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chsg8\" (UniqueName: \"kubernetes.io/projected/ed378475-ec00-4728-8580-8fc7cd8a57bc-kube-api-access-chsg8\") pod \"maas-api-revoke-sub-c5f3e839-5sb6l\" (UID: \"ed378475-ec00-4728-8580-8fc7cd8a57bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:53.961519 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.961485    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5f3e839-5sb6l\" (UID: \"ed378475-ec00-4728-8580-8fc7cd8a57bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:53.965136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.964177    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5f3e839-5sb6l\" (UID: \"ed378475-ec00-4728-8580-8fc7cd8a57bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:53.984351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:53.984317    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chsg8\" (UniqueName: \"kubernetes.io/projected/ed378475-ec00-4728-8580-8fc7cd8a57bc-kube-api-access-chsg8\") pod \"maas-api-revoke-sub-c5f3e839-5sb6l\" (UID: \"ed378475-ec00-4728-8580-8fc7cd8a57bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:54.088740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:54.088659    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:54.401400 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:21:54.401358    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded378475_ec00_4728_8580_8fc7cd8a57bc.slice/crio-1b7db81db59b02ec89481537e558ff33975b9520bee654c4396698eba597ffc0 WatchSource:0}: Error finding container 1b7db81db59b02ec89481537e558ff33975b9520bee654c4396698eba597ffc0: Status 404 returned error can't find the container with id 1b7db81db59b02ec89481537e558ff33975b9520bee654c4396698eba597ffc0
Nov 02 08:21:54.401775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:54.401747    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"]
Nov 02 08:21:54.843386 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:54.843348    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l" event={"ID":"ed378475-ec00-4728-8580-8fc7cd8a57bc","Type":"ContainerStarted","Data":"1b7db81db59b02ec89481537e558ff33975b9520bee654c4396698eba597ffc0"}
Nov 02 08:21:55.854196 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:55.854160    4898 generic.go:356] "Generic (PLEG): container finished" podID="ed378475-ec00-4728-8580-8fc7cd8a57bc" containerID="893838fd42863099c00dbf7e7fef18842e24579c95c53ea9b37e124e2be0be8e" exitCode=0
Nov 02 08:21:55.854613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:55.854245    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l" event={"ID":"ed378475-ec00-4728-8580-8fc7cd8a57bc","Type":"ContainerDied","Data":"893838fd42863099c00dbf7e7fef18842e24579c95c53ea9b37e124e2be0be8e"}
Nov 02 08:21:57.071265 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.071228    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:57.227515 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.227475    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed378475-ec00-4728-8580-8fc7cd8a57bc-kube-api-access-chsg8\" (UniqueName: \"kubernetes.io/projected/ed378475-ec00-4728-8580-8fc7cd8a57bc-kube-api-access-chsg8\") pod \"ed378475-ec00-4728-8580-8fc7cd8a57bc\" (UID: \"ed378475-ec00-4728-8580-8fc7cd8a57bc\") "
Nov 02 08:21:57.227782 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.227759    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-xks-ca\") pod \"ed378475-ec00-4728-8580-8fc7cd8a57bc\" (UID: \"ed378475-ec00-4728-8580-8fc7cd8a57bc\") "
Nov 02 08:21:57.227845 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.227811    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-openshift-ca\") pod \"ed378475-ec00-4728-8580-8fc7cd8a57bc\" (UID: \"ed378475-ec00-4728-8580-8fc7cd8a57bc\") "
Nov 02 08:21:57.228245 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.228213    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-openshift-ca" pod "ed378475-ec00-4728-8580-8fc7cd8a57bc" (UID: "ed378475-ec00-4728-8580-8fc7cd8a57bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:21:57.228933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.228883    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:57.229822 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.229789    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed378475-ec00-4728-8580-8fc7cd8a57bc-kube-api-access-chsg8" pod "ed378475-ec00-4728-8580-8fc7cd8a57bc" (UID: "ed378475-ec00-4728-8580-8fc7cd8a57bc"). InnerVolumeSpecName "kube-api-access-chsg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:21:57.229938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.229837    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-xks-ca" pod "ed378475-ec00-4728-8580-8fc7cd8a57bc" (UID: "ed378475-ec00-4728-8580-8fc7cd8a57bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:21:57.330576 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.330538    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chsg8\" (UniqueName: \"kubernetes.io/projected/ed378475-ec00-4728-8580-8fc7cd8a57bc-kube-api-access-chsg8\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:57.330576 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.330573    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed378475-ec00-4728-8580-8fc7cd8a57bc-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:21:57.877415 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.877318    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"
Nov 02 08:21:57.877415 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.877346    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l" event={"ID":"ed378475-ec00-4728-8580-8fc7cd8a57bc","Type":"ContainerDied","Data":"1b7db81db59b02ec89481537e558ff33975b9520bee654c4396698eba597ffc0"}
Nov 02 08:21:57.877415 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:21:57.877413    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b7db81db59b02ec89481537e558ff33975b9520bee654c4396698eba597ffc0"
Nov 02 08:22:12.098237 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.098200    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"]
Nov 02 08:22:12.099029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.099010    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ed378475-ec00-4728-8580-8fc7cd8a57bc" containerName="revoke-keys"
Nov 02 08:22:12.099102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.099031    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed378475-ec00-4728-8580-8fc7cd8a57bc" containerName="revoke-keys"
Nov 02 08:22:12.099102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.099090    4898 cpu_manager.go:414] "containerMap: removing container" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerName="istio-proxy"
Nov 02 08:22:12.099102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.099095    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb2e6c5-145c-4448-8404-1f676175dc7e" containerName="istio-proxy"
Nov 02 08:22:12.099252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.099242    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed378475-ec00-4728-8580-8fc7cd8a57bc" containerName="revoke-keys"
Nov 02 08:22:12.103413 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.103382    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:12.121994 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.121959    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"]
Nov 02 08:22:12.128126 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.128099    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:22:12.128399 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.128383    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:22:12.129141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.129117    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:22:12.130468 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.130449    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:22:12.231683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.231643    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d19e83c-th7z9\" (UID: \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:12.231871 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.231690    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm7qr\" (UniqueName: \"kubernetes.io/projected/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-kube-api-access-sm7qr\") pod \"maas-api-revoke-sub-8d19e83c-th7z9\" (UID: \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:12.231978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.231926    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d19e83c-th7z9\" (UID: \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:12.333321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.333283    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d19e83c-th7z9\" (UID: \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:12.333500 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.333447    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d19e83c-th7z9\" (UID: \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:12.333500 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.333471    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm7qr\" (UniqueName: \"kubernetes.io/projected/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-kube-api-access-sm7qr\") pod \"maas-api-revoke-sub-8d19e83c-th7z9\" (UID: \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:12.334119 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.334093    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d19e83c-th7z9\" (UID: \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:12.335671 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.335652    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d19e83c-th7z9\" (UID: \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:12.357519 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.357443    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm7qr\" (UniqueName: \"kubernetes.io/projected/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-kube-api-access-sm7qr\") pod \"maas-api-revoke-sub-8d19e83c-th7z9\" (UID: \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:12.456492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.456184    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:12.800717 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:22:12.800685    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29106bce_b602_43d7_bfbb_bd7ad1f4dc99.slice/crio-27ac98cfaf5b3a370c5e52c935e56e0aa2ac7205cf4a1ed2a9140b9601508056 WatchSource:0}: Error finding container 27ac98cfaf5b3a370c5e52c935e56e0aa2ac7205cf4a1ed2a9140b9601508056: Status 404 returned error can't find the container with id 27ac98cfaf5b3a370c5e52c935e56e0aa2ac7205cf4a1ed2a9140b9601508056
Nov 02 08:22:12.802005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:12.801977    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"]
Nov 02 08:22:13.039854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:13.039818    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9" event={"ID":"29106bce-b602-43d7-bfbb-bd7ad1f4dc99","Type":"ContainerStarted","Data":"27ac98cfaf5b3a370c5e52c935e56e0aa2ac7205cf4a1ed2a9140b9601508056"}
Nov 02 08:22:14.055031 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:14.054938    4898 generic.go:356] "Generic (PLEG): container finished" podID="29106bce-b602-43d7-bfbb-bd7ad1f4dc99" containerID="5909188927c5761741d9b3dd384bb173fc593fd17be979d4af0ea17dc31c3eb4" exitCode=0
Nov 02 08:22:14.055444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:14.055028    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9" event={"ID":"29106bce-b602-43d7-bfbb-bd7ad1f4dc99","Type":"ContainerDied","Data":"5909188927c5761741d9b3dd384bb173fc593fd17be979d4af0ea17dc31c3eb4"}
Nov 02 08:22:14.555063 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:14.555031    4898 scope.go:122] "RemoveContainer" containerID="0305ed181b7ec9aeef00603f005368787f26269bb552f00d69f74eebcaabab92"
Nov 02 08:22:14.567591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:14.567562    4898 scope.go:122] "RemoveContainer" containerID="4c6fea93710bf9715bf423016daae67ff2aa55cf76f167b958dbb97875991f77"
Nov 02 08:22:14.577583 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:14.577559    4898 scope.go:122] "RemoveContainer" containerID="5afcf13148017328f650adb58d839fd5d66d5e86870d65ec7ee18dd48a5aec29"
Nov 02 08:22:14.587472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:14.587443    4898 scope.go:122] "RemoveContainer" containerID="e142593ce02195c5d85af2e58799f9f275ce23474fa02d83a3052429cef06bb1"
Nov 02 08:22:14.597105 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:14.597086    4898 scope.go:122] "RemoveContainer" containerID="04e56efba7dcb43894935d4f6d303b69b1915a2d17ca91124150dfd9dcf4549e"
Nov 02 08:22:14.606662 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:14.606638    4898 scope.go:122] "RemoveContainer" containerID="c8231558612bd56bd63e773c7d350ac13b45c6e9350e56fa1bda3f44d1dc9dd3"
Nov 02 08:22:14.990671 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:22:14.990633    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:22:15.276499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:15.276471    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:15.389956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:15.389918    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-kube-api-access-sm7qr\" (UniqueName: \"kubernetes.io/projected/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-kube-api-access-sm7qr\") pod \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\" (UID: \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\") "
Nov 02 08:22:15.390125 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:15.390035    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-openshift-ca\") pod \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\" (UID: \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\") "
Nov 02 08:22:15.390125 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:15.390060    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-xks-ca\") pod \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\" (UID: \"29106bce-b602-43d7-bfbb-bd7ad1f4dc99\") "
Nov 02 08:22:15.390418 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:15.390387    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-openshift-ca" pod "29106bce-b602-43d7-bfbb-bd7ad1f4dc99" (UID: "29106bce-b602-43d7-bfbb-bd7ad1f4dc99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:22:15.390760 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:15.390743    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:22:15.392172 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:15.392140    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-kube-api-access-sm7qr" pod "29106bce-b602-43d7-bfbb-bd7ad1f4dc99" (UID: "29106bce-b602-43d7-bfbb-bd7ad1f4dc99"). InnerVolumeSpecName "kube-api-access-sm7qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:15.392351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:15.392325    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-xks-ca" pod "29106bce-b602-43d7-bfbb-bd7ad1f4dc99" (UID: "29106bce-b602-43d7-bfbb-bd7ad1f4dc99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:22:15.493141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:15.493106    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm7qr\" (UniqueName: \"kubernetes.io/projected/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-kube-api-access-sm7qr\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:22:15.493141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:15.493142    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29106bce-b602-43d7-bfbb-bd7ad1f4dc99-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:22:16.076762 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:16.076728    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"
Nov 02 08:22:16.076762 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:16.076747    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9" event={"ID":"29106bce-b602-43d7-bfbb-bd7ad1f4dc99","Type":"ContainerDied","Data":"27ac98cfaf5b3a370c5e52c935e56e0aa2ac7205cf4a1ed2a9140b9601508056"}
Nov 02 08:22:16.077035 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:16.076795    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ac98cfaf5b3a370c5e52c935e56e0aa2ac7205cf4a1ed2a9140b9601508056"
Nov 02 08:22:23.660040 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:23.660006    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"]
Nov 02 08:22:23.694779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:23.694703    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"]
Nov 02 08:22:24.816166 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.816123    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"]
Nov 02 08:22:24.816994 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.816974    4898 cpu_manager.go:414] "containerMap: removing container" podUID="29106bce-b602-43d7-bfbb-bd7ad1f4dc99" containerName="revoke-keys"
Nov 02 08:22:24.816994 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.816993    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29106bce-b602-43d7-bfbb-bd7ad1f4dc99" containerName="revoke-keys"
Nov 02 08:22:24.817237 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.817225    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="29106bce-b602-43d7-bfbb-bd7ad1f4dc99" containerName="revoke-keys"
Nov 02 08:22:24.822178 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.822151    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:24.844354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.841109    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"]
Nov 02 08:22:24.844854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.844826    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:22:24.844999 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.844838    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:22:24.844999 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.844901    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:22:24.850796 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.850778    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:22:24.972183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.972132    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsg9v\" (UniqueName: \"kubernetes.io/projected/d63168dd-cf88-4fd3-b4a5-6e8b39089859-kube-api-access-zsg9v\") pod \"maas-api-revoke-sub-00f2dd91-kp46l\" (UID: \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:24.972485 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.972461    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00f2dd91-kp46l\" (UID: \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:24.972549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:24.972493    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00f2dd91-kp46l\" (UID: \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:25.074312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:25.074220    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00f2dd91-kp46l\" (UID: \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:25.074312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:25.074263    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00f2dd91-kp46l\" (UID: \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:25.074552 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:25.074383    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsg9v\" (UniqueName: \"kubernetes.io/projected/d63168dd-cf88-4fd3-b4a5-6e8b39089859-kube-api-access-zsg9v\") pod \"maas-api-revoke-sub-00f2dd91-kp46l\" (UID: \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:25.074955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:25.074905    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00f2dd91-kp46l\" (UID: \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:25.077017 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:25.076986    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00f2dd91-kp46l\" (UID: \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:25.097731 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:25.097702    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsg9v\" (UniqueName: \"kubernetes.io/projected/d63168dd-cf88-4fd3-b4a5-6e8b39089859-kube-api-access-zsg9v\") pod \"maas-api-revoke-sub-00f2dd91-kp46l\" (UID: \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:25.150508 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:25.150479    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:25.539293 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:22:25.539256    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63168dd_cf88_4fd3_b4a5_6e8b39089859.slice/crio-339a32d912d5bfa901437ef5653e7eed05c64c3a5c59d56e37c54ae9c999a572 WatchSource:0}: Error finding container 339a32d912d5bfa901437ef5653e7eed05c64c3a5c59d56e37c54ae9c999a572: Status 404 returned error can't find the container with id 339a32d912d5bfa901437ef5653e7eed05c64c3a5c59d56e37c54ae9c999a572
Nov 02 08:22:25.539761 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:25.539736    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"]
Nov 02 08:22:26.226452 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:26.226417    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l" event={"ID":"d63168dd-cf88-4fd3-b4a5-6e8b39089859","Type":"ContainerStarted","Data":"6e50e10225607f1ee60bbfa2fffbb617915a328d10d5cef4b0abac2880568279"}
Nov 02 08:22:26.226845 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:26.226467    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l" event={"ID":"d63168dd-cf88-4fd3-b4a5-6e8b39089859","Type":"ContainerStarted","Data":"339a32d912d5bfa901437ef5653e7eed05c64c3a5c59d56e37c54ae9c999a572"}
Nov 02 08:22:26.270572 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:26.270508    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l" podStartSLOduration=2.270489364 podStartE2EDuration="2.270489364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:22:26.266571522 +0000 UTC m=+3496.956898509" watchObservedRunningTime="2026-11-02 08:22:26.270489364 +0000 UTC m=+3496.960816329"
Nov 02 08:22:27.236337 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:27.236291    4898 generic.go:356] "Generic (PLEG): container finished" podID="d63168dd-cf88-4fd3-b4a5-6e8b39089859" containerID="6e50e10225607f1ee60bbfa2fffbb617915a328d10d5cef4b0abac2880568279" exitCode=0
Nov 02 08:22:27.236745 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:27.236367    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l" event={"ID":"d63168dd-cf88-4fd3-b4a5-6e8b39089859","Type":"ContainerDied","Data":"6e50e10225607f1ee60bbfa2fffbb617915a328d10d5cef4b0abac2880568279"}
Nov 02 08:22:28.438291 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:28.438263    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:28.548355 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:28.548260    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-openshift-ca\") pod \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\" (UID: \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\") "
Nov 02 08:22:28.548355 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:28.548310    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-xks-ca\") pod \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\" (UID: \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\") "
Nov 02 08:22:28.548623 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:28.548367    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63168dd-cf88-4fd3-b4a5-6e8b39089859-kube-api-access-zsg9v\" (UniqueName: \"kubernetes.io/projected/d63168dd-cf88-4fd3-b4a5-6e8b39089859-kube-api-access-zsg9v\") pod \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\" (UID: \"d63168dd-cf88-4fd3-b4a5-6e8b39089859\") "
Nov 02 08:22:28.548721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:28.548696    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-openshift-ca" pod "d63168dd-cf88-4fd3-b4a5-6e8b39089859" (UID: "d63168dd-cf88-4fd3-b4a5-6e8b39089859"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:22:28.549475 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:28.549402    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:22:28.551128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:28.550851    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-xks-ca" pod "d63168dd-cf88-4fd3-b4a5-6e8b39089859" (UID: "d63168dd-cf88-4fd3-b4a5-6e8b39089859"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:22:28.551538 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:28.551517    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63168dd-cf88-4fd3-b4a5-6e8b39089859-kube-api-access-zsg9v" pod "d63168dd-cf88-4fd3-b4a5-6e8b39089859" (UID: "d63168dd-cf88-4fd3-b4a5-6e8b39089859"). InnerVolumeSpecName "kube-api-access-zsg9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:28.652911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:28.652458    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63168dd-cf88-4fd3-b4a5-6e8b39089859-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:22:28.652911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:28.652498    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsg9v\" (UniqueName: \"kubernetes.io/projected/d63168dd-cf88-4fd3-b4a5-6e8b39089859-kube-api-access-zsg9v\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:22:28.667916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:28.663007    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp" podUID="77a4db1f-32da-42da-ab59-c959e7812faf" containerName="payload-processing" containerID="cri-o://598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434" gracePeriod=25
Nov 02 08:22:28.695763 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:28.695717    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq" podUID="ffef3aaf-476b-4037-a4ac-cfc3c89bfb40" containerName="payload-pre-processing" containerID="cri-o://17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60" gracePeriod=25
Nov 02 08:22:29.254618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:29.254587    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"
Nov 02 08:22:29.254800 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:29.254626    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l" event={"ID":"d63168dd-cf88-4fd3-b4a5-6e8b39089859","Type":"ContainerDied","Data":"339a32d912d5bfa901437ef5653e7eed05c64c3a5c59d56e37c54ae9c999a572"}
Nov 02 08:22:29.254800 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:29.254675    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="339a32d912d5bfa901437ef5653e7eed05c64c3a5c59d56e37c54ae9c999a572"
Nov 02 08:22:36.331314 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:36.331264    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq" podUID="ffef3aaf-476b-4037-a4ac-cfc3c89bfb40" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:22:37.327021 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:37.326976    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp" podUID="77a4db1f-32da-42da-ab59-c959e7812faf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:22:44.995773 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:22:44.995725    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:22:46.318262 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:46.318222    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq" podUID="ffef3aaf-476b-4037-a4ac-cfc3c89bfb40" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:22:47.326417 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:47.326367    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp" podUID="77a4db1f-32da-42da-ab59-c959e7812faf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:22:48.904930 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:48.904463    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"
Nov 02 08:22:49.031382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.031353    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"
Nov 02 08:22:49.045616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.045586    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a4db1f-32da-42da-ab59-c959e7812faf-kube-api-access-lx4mk\" (UniqueName: \"kubernetes.io/projected/77a4db1f-32da-42da-ab59-c959e7812faf-kube-api-access-lx4mk\") pod \"77a4db1f-32da-42da-ab59-c959e7812faf\" (UID: \"77a4db1f-32da-42da-ab59-c959e7812faf\") "
Nov 02 08:22:49.045775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.045687    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a4db1f-32da-42da-ab59-c959e7812faf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a4db1f-32da-42da-ab59-c959e7812faf-plugins-config-volume\") pod \"77a4db1f-32da-42da-ab59-c959e7812faf\" (UID: \"77a4db1f-32da-42da-ab59-c959e7812faf\") "
Nov 02 08:22:49.046148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.046121    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a4db1f-32da-42da-ab59-c959e7812faf-plugins-config-volume" pod "77a4db1f-32da-42da-ab59-c959e7812faf" (UID: "77a4db1f-32da-42da-ab59-c959e7812faf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:22:49.046479 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.046461    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a4db1f-32da-42da-ab59-c959e7812faf-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:22:49.047701 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.047677    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a4db1f-32da-42da-ab59-c959e7812faf-kube-api-access-lx4mk" pod "77a4db1f-32da-42da-ab59-c959e7812faf" (UID: "77a4db1f-32da-42da-ab59-c959e7812faf"). InnerVolumeSpecName "kube-api-access-lx4mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:49.147670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.147638    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-plugins-config-volume\") pod \"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40\" (UID: \"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40\") "
Nov 02 08:22:49.147875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.147690    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-kube-api-access-cfht7\" (UniqueName: \"kubernetes.io/projected/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-kube-api-access-cfht7\") pod \"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40\" (UID: \"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40\") "
Nov 02 08:22:49.148145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.148119    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-plugins-config-volume" pod "ffef3aaf-476b-4037-a4ac-cfc3c89bfb40" (UID: "ffef3aaf-476b-4037-a4ac-cfc3c89bfb40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:22:49.148419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.148394    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:22:49.148419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.148415    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx4mk\" (UniqueName: \"kubernetes.io/projected/77a4db1f-32da-42da-ab59-c959e7812faf-kube-api-access-lx4mk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:22:49.149883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.149866    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-kube-api-access-cfht7" pod "ffef3aaf-476b-4037-a4ac-cfc3c89bfb40" (UID: "ffef3aaf-476b-4037-a4ac-cfc3c89bfb40"). InnerVolumeSpecName "kube-api-access-cfht7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:49.249942 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.249908    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfht7\" (UniqueName: \"kubernetes.io/projected/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40-kube-api-access-cfht7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:22:49.662448 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.662414    4898 generic.go:356] "Generic (PLEG): container finished" podID="77a4db1f-32da-42da-ab59-c959e7812faf" containerID="598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434" exitCode=0
Nov 02 08:22:49.662642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.662463    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"
Nov 02 08:22:49.662642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.662493    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp" event={"ID":"77a4db1f-32da-42da-ab59-c959e7812faf","Type":"ContainerDied","Data":"598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434"}
Nov 02 08:22:49.662642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.662562    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp" event={"ID":"77a4db1f-32da-42da-ab59-c959e7812faf","Type":"ContainerDied","Data":"e0d4b39217475833331898831de917a489309d2fd2412a96aaa8a9ac95ab348e"}
Nov 02 08:22:49.662642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.662590    4898 scope.go:122] "RemoveContainer" containerID="598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434"
Nov 02 08:22:49.665308 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.664917    4898 generic.go:356] "Generic (PLEG): container finished" podID="ffef3aaf-476b-4037-a4ac-cfc3c89bfb40" containerID="17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60" exitCode=0
Nov 02 08:22:49.665308 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.665014    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq" event={"ID":"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40","Type":"ContainerDied","Data":"17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60"}
Nov 02 08:22:49.665308 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.665053    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"
Nov 02 08:22:49.665567 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.665055    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq" event={"ID":"ffef3aaf-476b-4037-a4ac-cfc3c89bfb40","Type":"ContainerDied","Data":"a2db688565355e8e9f338e1c6616506b67dae4e3d59cd7c89f2602324361c495"}
Nov 02 08:22:49.675764 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.675742    4898 scope.go:122] "RemoveContainer" containerID="598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434"
Nov 02 08:22:49.676111 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:22:49.676086    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434\": container with ID starting with 598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434 not found: ID does not exist" containerID="598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434"
Nov 02 08:22:49.676285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.676257    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434"} err="failed to get container status \"598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434\": rpc error: code = NotFound desc = could not find container \"598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434\": container with ID starting with 598c0a25abb941fa5f4be29bef7b2655d43c60ce27f4f8f20fd9456f1c64f434 not found: ID does not exist"
Nov 02 08:22:49.676285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.676285    4898 scope.go:122] "RemoveContainer" containerID="17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60"
Nov 02 08:22:49.687512 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.687489    4898 scope.go:122] "RemoveContainer" containerID="17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60"
Nov 02 08:22:49.687815 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:22:49.687793    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60\": container with ID starting with 17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60 not found: ID does not exist" containerID="17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60"
Nov 02 08:22:49.687915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.687819    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60"} err="failed to get container status \"17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60\": rpc error: code = NotFound desc = could not find container \"17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60\": container with ID starting with 17d6a4c72a68a5887befe8fabaae36d3a7a1552ef9086e18a644c8c5cf6b6d60 not found: ID does not exist"
Nov 02 08:22:49.719204 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.719166    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"]
Nov 02 08:22:49.730611 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.730584    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e271b1-6f87f7d6b9-8gnlp"]
Nov 02 08:22:49.965932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:49.962947    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"]
Nov 02 08:22:50.008932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:50.007622    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e271b1-865cd788c5-c4gvq"]
Nov 02 08:22:51.470042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:51.469958    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a4db1f-32da-42da-ab59-c959e7812faf" path="/var/lib/kubelet/pods/77a4db1f-32da-42da-ab59-c959e7812faf/volumes"
Nov 02 08:22:51.470656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:51.470637    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffef3aaf-476b-4037-a4ac-cfc3c89bfb40" path="/var/lib/kubelet/pods/ffef3aaf-476b-4037-a4ac-cfc3c89bfb40/volumes"
Nov 02 08:22:52.986193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:52.986154    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"]
Nov 02 08:22:57.986960 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:57.986914    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42" podUID="c32626e9-75be-4a2b-b16b-3c5242be4989" containerName="payload-processing" containerID="cri-o://b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4" gracePeriod=25
Nov 02 08:22:58.303944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.303915    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"
Nov 02 08:22:58.446659 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.446629    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c32626e9-75be-4a2b-b16b-3c5242be4989-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32626e9-75be-4a2b-b16b-3c5242be4989-plugins-config-volume\") pod \"c32626e9-75be-4a2b-b16b-3c5242be4989\" (UID: \"c32626e9-75be-4a2b-b16b-3c5242be4989\") "
Nov 02 08:22:58.446834 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.446775    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c32626e9-75be-4a2b-b16b-3c5242be4989-kube-api-access-qqhqp\" (UniqueName: \"kubernetes.io/projected/c32626e9-75be-4a2b-b16b-3c5242be4989-kube-api-access-qqhqp\") pod \"c32626e9-75be-4a2b-b16b-3c5242be4989\" (UID: \"c32626e9-75be-4a2b-b16b-3c5242be4989\") "
Nov 02 08:22:58.447035 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.447014    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32626e9-75be-4a2b-b16b-3c5242be4989-plugins-config-volume" pod "c32626e9-75be-4a2b-b16b-3c5242be4989" (UID: "c32626e9-75be-4a2b-b16b-3c5242be4989"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:22:58.447301 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.447284    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32626e9-75be-4a2b-b16b-3c5242be4989-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:22:58.449102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.449073    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32626e9-75be-4a2b-b16b-3c5242be4989-kube-api-access-qqhqp" pod "c32626e9-75be-4a2b-b16b-3c5242be4989" (UID: "c32626e9-75be-4a2b-b16b-3c5242be4989"). InnerVolumeSpecName "kube-api-access-qqhqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:22:58.549268 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.549190    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqhqp\" (UniqueName: \"kubernetes.io/projected/c32626e9-75be-4a2b-b16b-3c5242be4989-kube-api-access-qqhqp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:22:58.810676 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.810585    4898 generic.go:356] "Generic (PLEG): container finished" podID="c32626e9-75be-4a2b-b16b-3c5242be4989" containerID="b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4" exitCode=0
Nov 02 08:22:58.810676 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.810636    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"
Nov 02 08:22:58.810883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.810674    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42" event={"ID":"c32626e9-75be-4a2b-b16b-3c5242be4989","Type":"ContainerDied","Data":"b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4"}
Nov 02 08:22:58.810883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.810732    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42" event={"ID":"c32626e9-75be-4a2b-b16b-3c5242be4989","Type":"ContainerDied","Data":"83010389436eff78bd1cfeba888beaa18eb8d1584c794ed514484dc8c9ea5d88"}
Nov 02 08:22:58.810883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.810754    4898 scope.go:122] "RemoveContainer" containerID="b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4"
Nov 02 08:22:58.822531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.822513    4898 scope.go:122] "RemoveContainer" containerID="b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4"
Nov 02 08:22:58.822796 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:22:58.822778    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4\": container with ID starting with b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4 not found: ID does not exist" containerID="b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4"
Nov 02 08:22:58.822840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.822803    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4"} err="failed to get container status \"b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4\": rpc error: code = NotFound desc = could not find container \"b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4\": container with ID starting with b8ae1d8747164c02dfd1189ef18c08afbc28712616159bcbd3817eee841835c4 not found: ID does not exist"
Nov 02 08:22:58.884270 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.884245    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"]
Nov 02 08:22:58.898799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.898774    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4489d-7864c48c5c-hhd42"]
Nov 02 08:22:58.987027 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:58.986993    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"]
Nov 02 08:22:59.469554 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:22:59.469511    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c32626e9-75be-4a2b-b16b-3c5242be4989" path="/var/lib/kubelet/pods/c32626e9-75be-4a2b-b16b-3c5242be4989/volumes"
Nov 02 08:23:03.997913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:03.990965    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j" podUID="30f79c15-0f1d-4041-858c-07252b72704e" containerName="payload-pre-processing" containerID="cri-o://e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597" gracePeriod=25
Nov 02 08:23:08.091356 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:08.091308    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"]
Nov 02 08:23:08.091812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:08.091573    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy" containerID="cri-o://4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14" gracePeriod=30
Nov 02 08:23:08.346962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:08.346042    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j" podUID="30f79c15-0f1d-4041-858c-07252b72704e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:23:13.406592 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.406561    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:23:13.461714 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.461677    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-certs\") pod \"cb263662-7712-4909-92c9-deb40f800844\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") "
Nov 02 08:23:13.461940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.461738    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb263662-7712-4909-92c9-deb40f800844-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb263662-7712-4909-92c9-deb40f800844-istiod-ca-cert\") pod \"cb263662-7712-4909-92c9-deb40f800844\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") "
Nov 02 08:23:13.461940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.461772    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-data\") pod \"cb263662-7712-4909-92c9-deb40f800844\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") "
Nov 02 08:23:13.461940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.461808    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-istio-token\" (UniqueName: \"kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-istio-token\") pod \"cb263662-7712-4909-92c9-deb40f800844\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") "
Nov 02 08:23:13.462132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.461938    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb263662-7712-4909-92c9-deb40f800844-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb263662-7712-4909-92c9-deb40f800844-istio-podinfo\") pod \"cb263662-7712-4909-92c9-deb40f800844\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") "
Nov 02 08:23:13.462132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.461984    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-kube-api-access-pcm68\" (UniqueName: \"kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-kube-api-access-pcm68\") pod \"cb263662-7712-4909-92c9-deb40f800844\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") "
Nov 02 08:23:13.462132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.461980    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-certs" pod "cb263662-7712-4909-92c9-deb40f800844" (UID: "cb263662-7712-4909-92c9-deb40f800844"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:23:13.462132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.462057    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-data" pod "cb263662-7712-4909-92c9-deb40f800844" (UID: "cb263662-7712-4909-92c9-deb40f800844"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:23:13.462367 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.462261    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb263662-7712-4909-92c9-deb40f800844-istiod-ca-cert" pod "cb263662-7712-4909-92c9-deb40f800844" (UID: "cb263662-7712-4909-92c9-deb40f800844"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:23:13.462427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.462382    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-socket\") pod \"cb263662-7712-4909-92c9-deb40f800844\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") "
Nov 02 08:23:13.462486 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.462462    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-envoy\") pod \"cb263662-7712-4909-92c9-deb40f800844\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") "
Nov 02 08:23:13.462708 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.462586    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-credential-socket\") pod \"cb263662-7712-4909-92c9-deb40f800844\" (UID: \"cb263662-7712-4909-92c9-deb40f800844\") "
Nov 02 08:23:13.462770 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.462701    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-socket" pod "cb263662-7712-4909-92c9-deb40f800844" (UID: "cb263662-7712-4909-92c9-deb40f800844"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:23:13.462835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.462811    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-credential-socket" pod "cb263662-7712-4909-92c9-deb40f800844" (UID: "cb263662-7712-4909-92c9-deb40f800844"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:23:13.463352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.463332    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:13.463421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.463355    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:13.463421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.463369    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb263662-7712-4909-92c9-deb40f800844-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:13.463421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.463382    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:13.463421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.463395    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:13.465928 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.464792    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb263662-7712-4909-92c9-deb40f800844-istio-podinfo" pod "cb263662-7712-4909-92c9-deb40f800844" (UID: "cb263662-7712-4909-92c9-deb40f800844"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:23:13.466079 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.465947    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-envoy" pod "cb263662-7712-4909-92c9-deb40f800844" (UID: "cb263662-7712-4909-92c9-deb40f800844"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:23:13.466079 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.465960    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-istio-token" pod "cb263662-7712-4909-92c9-deb40f800844" (UID: "cb263662-7712-4909-92c9-deb40f800844"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:23:13.466276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.466254    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-kube-api-access-pcm68" pod "cb263662-7712-4909-92c9-deb40f800844" (UID: "cb263662-7712-4909-92c9-deb40f800844"). InnerVolumeSpecName "kube-api-access-pcm68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:23:13.566013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.565973    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb263662-7712-4909-92c9-deb40f800844-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:13.566013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.566012    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:13.566214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.566028    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb263662-7712-4909-92c9-deb40f800844-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:13.566214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:13.566041    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcm68\" (UniqueName: \"kubernetes.io/projected/cb263662-7712-4909-92c9-deb40f800844-kube-api-access-pcm68\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:14.001946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.001903    4898 generic.go:356] "Generic (PLEG): container finished" podID="cb263662-7712-4909-92c9-deb40f800844" containerID="4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14" exitCode=0
Nov 02 08:23:14.002145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.001944    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" event={"ID":"cb263662-7712-4909-92c9-deb40f800844","Type":"ContainerDied","Data":"4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14"}
Nov 02 08:23:14.002145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.001974    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"
Nov 02 08:23:14.002145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.002013    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6" event={"ID":"cb263662-7712-4909-92c9-deb40f800844","Type":"ContainerDied","Data":"535b2819c0f4465b213febefcbbb0684c2215776d9ddd9462b27be2b74060e82"}
Nov 02 08:23:14.002145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.002030    4898 scope.go:122] "RemoveContainer" containerID="4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14"
Nov 02 08:23:14.015083 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.015063    4898 scope.go:122] "RemoveContainer" containerID="d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6"
Nov 02 08:23:14.028820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.028796    4898 scope.go:122] "RemoveContainer" containerID="4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14"
Nov 02 08:23:14.029110 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:23:14.029089    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14\": container with ID starting with 4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14 not found: ID does not exist" containerID="4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14"
Nov 02 08:23:14.029191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.029118    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14"} err="failed to get container status \"4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14\": rpc error: code = NotFound desc = could not find container \"4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14\": container with ID starting with 4da1705d0251e0769caf7095a8cdf0b137b673f10788d0a246eb0d6b25353b14 not found: ID does not exist"
Nov 02 08:23:14.029191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.029138    4898 scope.go:122] "RemoveContainer" containerID="d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6"
Nov 02 08:23:14.029430 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:23:14.029411    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6\": container with ID starting with d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6 not found: ID does not exist" containerID="d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6"
Nov 02 08:23:14.029476 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.029436    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6"} err="failed to get container status \"d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6\": rpc error: code = NotFound desc = could not find container \"d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6\": container with ID starting with d640de4933d6584940ad0a5a222afd9ccb1af8822a7418f391fabfab01bf4fc6 not found: ID does not exist"
Nov 02 08:23:14.063957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.063931    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"]
Nov 02 08:23:14.071691 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.071666    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f765b8-openshift-default-d9b984d69-4ncc6"]
Nov 02 08:23:14.683218 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.683180    4898 scope.go:122] "RemoveContainer" containerID="4d888e4dae980ea3e79cd70082333476e6807aa29d040cc8d03f1f214c6c377f"
Nov 02 08:23:14.697219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:14.697198    4898 scope.go:122] "RemoveContainer" containerID="8af6351d26962b9654d028842bd87f18b65e75ace524849183dbeed0012f8aff"
Nov 02 08:23:14.993156 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:23:14.993063    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:23:15.469287 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:15.469250    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb263662-7712-4909-92c9-deb40f800844" path="/var/lib/kubelet/pods/cb263662-7712-4909-92c9-deb40f800844/volumes"
Nov 02 08:23:18.297059 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:18.297015    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j" podUID="30f79c15-0f1d-4041-858c-07252b72704e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:23:24.348201 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:24.348158    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"
Nov 02 08:23:24.380252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:24.380210    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f79c15-0f1d-4041-858c-07252b72704e-kube-api-access-fvqjb\" (UniqueName: \"kubernetes.io/projected/30f79c15-0f1d-4041-858c-07252b72704e-kube-api-access-fvqjb\") pod \"30f79c15-0f1d-4041-858c-07252b72704e\" (UID: \"30f79c15-0f1d-4041-858c-07252b72704e\") "
Nov 02 08:23:24.380498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:24.380465    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f79c15-0f1d-4041-858c-07252b72704e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f79c15-0f1d-4041-858c-07252b72704e-plugins-config-volume\") pod \"30f79c15-0f1d-4041-858c-07252b72704e\" (UID: \"30f79c15-0f1d-4041-858c-07252b72704e\") "
Nov 02 08:23:24.380884 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:24.380854    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f79c15-0f1d-4041-858c-07252b72704e-plugins-config-volume" pod "30f79c15-0f1d-4041-858c-07252b72704e" (UID: "30f79c15-0f1d-4041-858c-07252b72704e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:23:24.381115 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:24.381089    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f79c15-0f1d-4041-858c-07252b72704e-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:24.382645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:24.382616    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f79c15-0f1d-4041-858c-07252b72704e-kube-api-access-fvqjb" pod "30f79c15-0f1d-4041-858c-07252b72704e" (UID: "30f79c15-0f1d-4041-858c-07252b72704e"). InnerVolumeSpecName "kube-api-access-fvqjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:23:24.482695 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:24.482661    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvqjb\" (UniqueName: \"kubernetes.io/projected/30f79c15-0f1d-4041-858c-07252b72704e-kube-api-access-fvqjb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:25.144370 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:25.144332    4898 generic.go:356] "Generic (PLEG): container finished" podID="30f79c15-0f1d-4041-858c-07252b72704e" containerID="e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597" exitCode=0
Nov 02 08:23:25.144583 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:25.144384    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"
Nov 02 08:23:25.144583 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:25.144419    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j" event={"ID":"30f79c15-0f1d-4041-858c-07252b72704e","Type":"ContainerDied","Data":"e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597"}
Nov 02 08:23:25.144583 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:25.144492    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j" event={"ID":"30f79c15-0f1d-4041-858c-07252b72704e","Type":"ContainerDied","Data":"2543b9787772195fa284d66c9f1d6de8529773bfa535515f88ecdc90972262e9"}
Nov 02 08:23:25.144583 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:25.144522    4898 scope.go:122] "RemoveContainer" containerID="e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597"
Nov 02 08:23:25.156727 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:25.156704    4898 scope.go:122] "RemoveContainer" containerID="e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597"
Nov 02 08:23:25.157028 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:23:25.157008    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597\": container with ID starting with e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597 not found: ID does not exist" containerID="e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597"
Nov 02 08:23:25.157084 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:25.157036    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597"} err="failed to get container status \"e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597\": rpc error: code = NotFound desc = could not find container \"e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597\": container with ID starting with e0b2dd051ea800698e99aeebbb016ecbb4d4ff0097526873d205c48cb7ae2597 not found: ID does not exist"
Nov 02 08:23:25.202329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:25.202299    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"]
Nov 02 08:23:25.214121 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:25.214089    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f765b8-d59ccb87-t5g7j"]
Nov 02 08:23:25.466738 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:25.466701    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f79c15-0f1d-4041-858c-07252b72704e" path="/var/lib/kubelet/pods/30f79c15-0f1d-4041-858c-07252b72704e/volumes"
Nov 02 08:23:40.602618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:40.602584    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"]
Nov 02 08:23:40.603178 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:40.602863    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy" containerID="cri-o://6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c" gracePeriod=30
Nov 02 08:23:44.990443 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:23:44.990398    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:23:45.897662 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.897632    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:23:45.985850 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.985798    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2895ef-71fb-4097-829d-bfb60039769f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2895ef-71fb-4097-829d-bfb60039769f-istiod-ca-cert\") pod \"0d2895ef-71fb-4097-829d-bfb60039769f\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") "
Nov 02 08:23:45.986105 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.985955    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-data\") pod \"0d2895ef-71fb-4097-829d-bfb60039769f\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") "
Nov 02 08:23:45.986105 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.985996    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-credential-socket\") pod \"0d2895ef-71fb-4097-829d-bfb60039769f\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") "
Nov 02 08:23:45.986250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.986142    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-certs\") pod \"0d2895ef-71fb-4097-829d-bfb60039769f\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") "
Nov 02 08:23:45.986250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.986198    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d2895ef-71fb-4097-829d-bfb60039769f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2895ef-71fb-4097-829d-bfb60039769f-istio-podinfo\") pod \"0d2895ef-71fb-4097-829d-bfb60039769f\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") "
Nov 02 08:23:45.986250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.986207    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-data" pod "0d2895ef-71fb-4097-829d-bfb60039769f" (UID: "0d2895ef-71fb-4097-829d-bfb60039769f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:23:45.986250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.986213    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2895ef-71fb-4097-829d-bfb60039769f-istiod-ca-cert" pod "0d2895ef-71fb-4097-829d-bfb60039769f" (UID: "0d2895ef-71fb-4097-829d-bfb60039769f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:23:45.986250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.986239    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-istio-token\" (UniqueName: \"kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-istio-token\") pod \"0d2895ef-71fb-4097-829d-bfb60039769f\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") "
Nov 02 08:23:45.986549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.986342    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-credential-socket" pod "0d2895ef-71fb-4097-829d-bfb60039769f" (UID: "0d2895ef-71fb-4097-829d-bfb60039769f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:23:45.986549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.986374    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-envoy\") pod \"0d2895ef-71fb-4097-829d-bfb60039769f\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") "
Nov 02 08:23:45.986549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.986443    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-kube-api-access-fhr9n\" (UniqueName: \"kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-kube-api-access-fhr9n\") pod \"0d2895ef-71fb-4097-829d-bfb60039769f\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") "
Nov 02 08:23:45.986549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.986495    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-certs" pod "0d2895ef-71fb-4097-829d-bfb60039769f" (UID: "0d2895ef-71fb-4097-829d-bfb60039769f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:23:45.986549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.986524    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-socket\") pod \"0d2895ef-71fb-4097-829d-bfb60039769f\" (UID: \"0d2895ef-71fb-4097-829d-bfb60039769f\") "
Nov 02 08:23:45.986854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.986829    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-socket" pod "0d2895ef-71fb-4097-829d-bfb60039769f" (UID: "0d2895ef-71fb-4097-829d-bfb60039769f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:23:45.987569 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.987534    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:45.987569 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.987564    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:45.987779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.987577    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2895ef-71fb-4097-829d-bfb60039769f-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:45.987779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.987592    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:45.987779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.987604    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:45.988971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.988942    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d2895ef-71fb-4097-829d-bfb60039769f-istio-podinfo" pod "0d2895ef-71fb-4097-829d-bfb60039769f" (UID: "0d2895ef-71fb-4097-829d-bfb60039769f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:23:45.989107 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.989031    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-kube-api-access-fhr9n" pod "0d2895ef-71fb-4097-829d-bfb60039769f" (UID: "0d2895ef-71fb-4097-829d-bfb60039769f"). InnerVolumeSpecName "kube-api-access-fhr9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:23:45.989185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.989172    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-envoy" pod "0d2895ef-71fb-4097-829d-bfb60039769f" (UID: "0d2895ef-71fb-4097-829d-bfb60039769f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:23:45.989366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:45.989348    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-istio-token" pod "0d2895ef-71fb-4097-829d-bfb60039769f" (UID: "0d2895ef-71fb-4097-829d-bfb60039769f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:23:46.088946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.088861    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2895ef-71fb-4097-829d-bfb60039769f-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:46.088946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.088932    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:46.088946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.088942    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2895ef-71fb-4097-829d-bfb60039769f-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:46.088946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.088951    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhr9n\" (UniqueName: \"kubernetes.io/projected/0d2895ef-71fb-4097-829d-bfb60039769f-kube-api-access-fhr9n\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:23:46.349077 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.349043    4898 generic.go:356] "Generic (PLEG): container finished" podID="0d2895ef-71fb-4097-829d-bfb60039769f" containerID="6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c" exitCode=0
Nov 02 08:23:46.349290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.349096    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"
Nov 02 08:23:46.349290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.349133    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" event={"ID":"0d2895ef-71fb-4097-829d-bfb60039769f","Type":"ContainerDied","Data":"6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c"}
Nov 02 08:23:46.349290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.349203    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6" event={"ID":"0d2895ef-71fb-4097-829d-bfb60039769f","Type":"ContainerDied","Data":"ab49b12642b4ea1d666428a1d2df354bed0f523e6740a2f38e80268d95a87475"}
Nov 02 08:23:46.349290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.349235    4898 scope.go:122] "RemoveContainer" containerID="6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c"
Nov 02 08:23:46.361659 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.361640    4898 scope.go:122] "RemoveContainer" containerID="6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116"
Nov 02 08:23:46.372063 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.372042    4898 scope.go:122] "RemoveContainer" containerID="6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c"
Nov 02 08:23:46.372364 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:23:46.372343    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c\": container with ID starting with 6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c not found: ID does not exist" containerID="6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c"
Nov 02 08:23:46.372453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.372375    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c"} err="failed to get container status \"6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c\": rpc error: code = NotFound desc = could not find container \"6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c\": container with ID starting with 6b1f21f27b8f4655fa94f2407cc2004a3a71a7b51d55c67bb485b3fc0709260c not found: ID does not exist"
Nov 02 08:23:46.372453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.372398    4898 scope.go:122] "RemoveContainer" containerID="6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116"
Nov 02 08:23:46.372643 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:23:46.372624    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116\": container with ID starting with 6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116 not found: ID does not exist" containerID="6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116"
Nov 02 08:23:46.372689 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.372649    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116"} err="failed to get container status \"6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116\": rpc error: code = NotFound desc = could not find container \"6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116\": container with ID starting with 6a303253866c8f881f0106cdcc79601ee8f35253c24dea933336e47cbbe77116 not found: ID does not exist"
Nov 02 08:23:46.408617 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.408583    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"]
Nov 02 08:23:46.419597 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:46.419563    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c4489d-openshift-default-74d45cccd7-zs9q6"]
Nov 02 08:23:47.467361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:23:47.467325    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" path="/var/lib/kubelet/pods/0d2895ef-71fb-4097-829d-bfb60039769f/volumes"
Nov 02 08:24:10.684660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:10.684609    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:24:10.689709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:10.689682    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:24:14.994544 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:24:14.994498    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:24:41.269227 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.269183    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"]
Nov 02 08:24:41.270528 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270497    4898 cpu_manager.go:414] "containerMap: removing container" podUID="77a4db1f-32da-42da-ab59-c959e7812faf" containerName="payload-processing"
Nov 02 08:24:41.270528 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270525    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a4db1f-32da-42da-ab59-c959e7812faf" containerName="payload-processing"
Nov 02 08:24:41.270700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270541    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy"
Nov 02 08:24:41.270700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270549    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy"
Nov 02 08:24:41.270700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270581    4898 cpu_manager.go:414] "containerMap: removing container" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy"
Nov 02 08:24:41.270700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270590    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy"
Nov 02 08:24:41.270700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270605    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ffef3aaf-476b-4037-a4ac-cfc3c89bfb40" containerName="payload-pre-processing"
Nov 02 08:24:41.270700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270613    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffef3aaf-476b-4037-a4ac-cfc3c89bfb40" containerName="payload-pre-processing"
Nov 02 08:24:41.270700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270661    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c32626e9-75be-4a2b-b16b-3c5242be4989" containerName="payload-processing"
Nov 02 08:24:41.270700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270669    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32626e9-75be-4a2b-b16b-3c5242be4989" containerName="payload-processing"
Nov 02 08:24:41.270984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270712    4898 cpu_manager.go:414] "containerMap: removing container" podUID="30f79c15-0f1d-4041-858c-07252b72704e" containerName="payload-pre-processing"
Nov 02 08:24:41.270984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270721    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f79c15-0f1d-4041-858c-07252b72704e" containerName="payload-pre-processing"
Nov 02 08:24:41.270984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270734    4898 cpu_manager.go:414] "containerMap: removing container" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy"
Nov 02 08:24:41.270984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270741    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy"
Nov 02 08:24:41.270984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270757    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d63168dd-cf88-4fd3-b4a5-6e8b39089859" containerName="revoke-keys"
Nov 02 08:24:41.270984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270764    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63168dd-cf88-4fd3-b4a5-6e8b39089859" containerName="revoke-keys"
Nov 02 08:24:41.270984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270955    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63168dd-cf88-4fd3-b4a5-6e8b39089859" containerName="revoke-keys"
Nov 02 08:24:41.270984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270977    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a4db1f-32da-42da-ab59-c959e7812faf" containerName="payload-processing"
Nov 02 08:24:41.271240 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.270987    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy"
Nov 02 08:24:41.271240 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.271002    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy"
Nov 02 08:24:41.271240 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.271013    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffef3aaf-476b-4037-a4ac-cfc3c89bfb40" containerName="payload-pre-processing"
Nov 02 08:24:41.271240 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.271026    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy"
Nov 02 08:24:41.271240 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.271039    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32626e9-75be-4a2b-b16b-3c5242be4989" containerName="payload-processing"
Nov 02 08:24:41.271240 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.271052    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f79c15-0f1d-4041-858c-07252b72704e" containerName="payload-pre-processing"
Nov 02 08:24:41.278309 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.278287    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.328332 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.328289    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.328597 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.328576    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.328765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.328748    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7p6\" (UniqueName: \"kubernetes.io/projected/12755c83-2699-430b-8342-9c06b8273215-kube-api-access-nt7p6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.328988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.328972    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.329151 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.329137    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12755c83-2699-430b-8342-9c06b8273215-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.329291 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.329276    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12755c83-2699-430b-8342-9c06b8273215-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.329441 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.329428    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.329611 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.329598    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12755c83-2699-430b-8342-9c06b8273215-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.329788 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.329768    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.369244 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.369211    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"]
Nov 02 08:24:41.431375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.431336    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.431553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.431389    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.431553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.431414    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7p6\" (UniqueName: \"kubernetes.io/projected/12755c83-2699-430b-8342-9c06b8273215-kube-api-access-nt7p6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.431553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.431473    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.431553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.431501    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12755c83-2699-430b-8342-9c06b8273215-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.431553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.431526    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12755c83-2699-430b-8342-9c06b8273215-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.431840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.431595    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.431840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.431700    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12755c83-2699-430b-8342-9c06b8273215-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.431840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.431748    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.431840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.431830    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.432079 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.431865    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.432079 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.431938    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.432201 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.432183    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.432566 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.432543    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12755c83-2699-430b-8342-9c06b8273215-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.433969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.433945    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12755c83-2699-430b-8342-9c06b8273215-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.434217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.434196    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12755c83-2699-430b-8342-9c06b8273215-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.455013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.454984    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12755c83-2699-430b-8342-9c06b8273215-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.455232 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.455207    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7p6\" (UniqueName: \"kubernetes.io/projected/12755c83-2699-430b-8342-9c06b8273215-kube-api-access-nt7p6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jxxsm\" (UID: \"12755c83-2699-430b-8342-9c06b8273215\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.602299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.602225    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:41.932173 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:24:41.932132    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12755c83_2699_430b_8342_9c06b8273215.slice/crio-4718778062a539d7217fc0d7466888e5d28c60fe6cc498c95fd42a462bcc2ae6 WatchSource:0}: Error finding container 4718778062a539d7217fc0d7466888e5d28c60fe6cc498c95fd42a462bcc2ae6: Status 404 returned error can't find the container with id 4718778062a539d7217fc0d7466888e5d28c60fe6cc498c95fd42a462bcc2ae6
Nov 02 08:24:41.935210 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.935173    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:24:41.935308 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.935244    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:24:41.935308 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.935274    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:24:41.940239 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.940217    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"]
Nov 02 08:24:41.971960 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:41.971930    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm" event={"ID":"12755c83-2699-430b-8342-9c06b8273215","Type":"ContainerStarted","Data":"4718778062a539d7217fc0d7466888e5d28c60fe6cc498c95fd42a462bcc2ae6"}
Nov 02 08:24:42.992240 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:42.992161    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm" event={"ID":"12755c83-2699-430b-8342-9c06b8273215","Type":"ContainerStarted","Data":"4009cf749635df1514f020629638eaa4f6ce61a7a01ecd8c5b666681b298fccc"}
Nov 02 08:24:43.041907 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.041847    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm" podStartSLOduration=2.041832456 podStartE2EDuration="2.041832456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:24:43.041482017 +0000 UTC m=+3633.731809001" watchObservedRunningTime="2026-11-02 08:24:43.041832456 +0000 UTC m=+3633.732159420"
Nov 02 08:24:43.434130 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.434093    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"]
Nov 02 08:24:43.434927 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.434882    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy"
Nov 02 08:24:43.434927 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.434917    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2895ef-71fb-4097-829d-bfb60039769f" containerName="istio-proxy"
Nov 02 08:24:43.435171 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.435147    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb263662-7712-4909-92c9-deb40f800844" containerName="istio-proxy"
Nov 02 08:24:43.439065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.439047    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:43.454129 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.454104    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:24:43.454577 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.454557    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:24:43.454651 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.454600    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:24:43.454693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.454686    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:24:43.458908 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.457507    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"]
Nov 02 08:24:43.600256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.600221    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f8bc9d-8fb58\" (UID: \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:43.600423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.600356    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndtmf\" (UniqueName: \"kubernetes.io/projected/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-kube-api-access-ndtmf\") pod \"maas-api-revoke-sub-f0f8bc9d-8fb58\" (UID: \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:43.600489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.600461    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f8bc9d-8fb58\" (UID: \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:43.602843 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.602821    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:43.604098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.604074    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jxxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 02 08:24:43.604223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.604119    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm" podUID="12755c83-2699-430b-8342-9c06b8273215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 02 08:24:43.702564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.702471    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f8bc9d-8fb58\" (UID: \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:43.702729 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.702601    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndtmf\" (UniqueName: \"kubernetes.io/projected/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-kube-api-access-ndtmf\") pod \"maas-api-revoke-sub-f0f8bc9d-8fb58\" (UID: \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:43.702729 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.702657    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f8bc9d-8fb58\" (UID: \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:43.703303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.703277    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f8bc9d-8fb58\" (UID: \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:43.705337 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.705312    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f8bc9d-8fb58\" (UID: \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:43.726143 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.726111    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndtmf\" (UniqueName: \"kubernetes.io/projected/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-kube-api-access-ndtmf\") pod \"maas-api-revoke-sub-f0f8bc9d-8fb58\" (UID: \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:43.765933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:43.764701    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:44.120564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:44.120529    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"]
Nov 02 08:24:44.120987 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:24:44.120762    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3ec7ac_934c_4a5b_ab71_7e63615c34b7.slice/crio-374ddd04cbeabb4f46f1535e1abf48d3c45bacbb3b13a6e70820fabcc5b95702 WatchSource:0}: Error finding container 374ddd04cbeabb4f46f1535e1abf48d3c45bacbb3b13a6e70820fabcc5b95702: Status 404 returned error can't find the container with id 374ddd04cbeabb4f46f1535e1abf48d3c45bacbb3b13a6e70820fabcc5b95702
Nov 02 08:24:44.603105 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:44.603064    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jxxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 02 08:24:44.603299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:44.603133    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm" podUID="12755c83-2699-430b-8342-9c06b8273215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 02 08:24:45.013164 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:24:45.013063    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:24:45.018223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:45.018165    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58" event={"ID":"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7","Type":"ContainerStarted","Data":"25b73451ab794f5cac100ff54c5ac52943f4f974201ddd105c5a5ab52f0c6261"}
Nov 02 08:24:45.018387 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:45.018240    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58" event={"ID":"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7","Type":"ContainerStarted","Data":"374ddd04cbeabb4f46f1535e1abf48d3c45bacbb3b13a6e70820fabcc5b95702"}
Nov 02 08:24:45.092667 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:45.092608    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58" podStartSLOduration=2.092591573 podStartE2EDuration="2.092591573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:24:45.058371158 +0000 UTC m=+3635.748698140" watchObservedRunningTime="2026-11-02 08:24:45.092591573 +0000 UTC m=+3635.782918534"
Nov 02 08:24:45.603384 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:45.603348    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jxxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 02 08:24:45.603854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:45.603406    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm" podUID="12755c83-2699-430b-8342-9c06b8273215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 02 08:24:46.029223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:46.029184    4898 generic.go:356] "Generic (PLEG): container finished" podID="0a3ec7ac-934c-4a5b-ab71-7e63615c34b7" containerID="25b73451ab794f5cac100ff54c5ac52943f4f974201ddd105c5a5ab52f0c6261" exitCode=0
Nov 02 08:24:46.029427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:46.029264    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58" event={"ID":"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7","Type":"ContainerDied","Data":"25b73451ab794f5cac100ff54c5ac52943f4f974201ddd105c5a5ab52f0c6261"}
Nov 02 08:24:46.603317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:46.603282    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jxxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 02 08:24:46.603540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:46.603343    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm" podUID="12755c83-2699-430b-8342-9c06b8273215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 02 08:24:47.411975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.411942    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"]
Nov 02 08:24:47.412428 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.412396    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy" containerID="cri-o://eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1" gracePeriod=30
Nov 02 08:24:47.419378 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.419326    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:47.631033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.627984    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jxxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 02 08:24:47.631033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.628047    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm" podUID="12755c83-2699-430b-8342-9c06b8273215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 02 08:24:47.631033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.630653    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-kube-api-access-ndtmf\" (UniqueName: \"kubernetes.io/projected/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-kube-api-access-ndtmf\") pod \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\" (UID: \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\") "
Nov 02 08:24:47.631033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.630727    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-xks-ca\") pod \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\" (UID: \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\") "
Nov 02 08:24:47.631033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.630780    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-openshift-ca\") pod \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\" (UID: \"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7\") "
Nov 02 08:24:47.644923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.642212    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-openshift-ca" pod "0a3ec7ac-934c-4a5b-ab71-7e63615c34b7" (UID: "0a3ec7ac-934c-4a5b-ab71-7e63615c34b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:24:47.684089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.667008    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-xks-ca" pod "0a3ec7ac-934c-4a5b-ab71-7e63615c34b7" (UID: "0a3ec7ac-934c-4a5b-ab71-7e63615c34b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:24:47.684089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.678473    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-kube-api-access-ndtmf" pod "0a3ec7ac-934c-4a5b-ab71-7e63615c34b7" (UID: "0a3ec7ac-934c-4a5b-ab71-7e63615c34b7"). InnerVolumeSpecName "kube-api-access-ndtmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:24:47.699964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.699229    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndtmf\" (UniqueName: \"kubernetes.io/projected/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-kube-api-access-ndtmf\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:24:47.699964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.699284    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:24:47.699964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:47.699307    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:24:48.046903 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:48.046859    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"
Nov 02 08:24:48.047117 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:48.046904    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58" event={"ID":"0a3ec7ac-934c-4a5b-ab71-7e63615c34b7","Type":"ContainerDied","Data":"374ddd04cbeabb4f46f1535e1abf48d3c45bacbb3b13a6e70820fabcc5b95702"}
Nov 02 08:24:48.047117 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:48.046948    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="374ddd04cbeabb4f46f1535e1abf48d3c45bacbb3b13a6e70820fabcc5b95702"
Nov 02 08:24:48.533522 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:48.533481    4898 patch_prober.go:29] interesting pod/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:24:48.533717 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:48.533548    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:24:48.603303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:48.603263    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jxxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 02 08:24:48.603487 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:48.603331    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm" podUID="12755c83-2699-430b-8342-9c06b8273215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 02 08:24:49.602783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:49.602743    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jxxsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Nov 02 08:24:49.603264 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:49.602812    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm" podUID="12755c83-2699-430b-8342-9c06b8273215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Nov 02 08:24:50.976210 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:50.976156    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:50.978932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:50.977011    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:50.979191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:50.979170    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm"
Nov 02 08:24:52.725876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.725843    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:24:52.811638 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.811593    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1900eb00-365d-401d-a399-946fb0c03bfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1900eb00-365d-401d-a399-946fb0c03bfa-istiod-ca-cert\") pod \"1900eb00-365d-401d-a399-946fb0c03bfa\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") "
Nov 02 08:24:52.812538 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.812503    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-data\") pod \"1900eb00-365d-401d-a399-946fb0c03bfa\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") "
Nov 02 08:24:52.813300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.813278    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-credential-socket\") pod \"1900eb00-365d-401d-a399-946fb0c03bfa\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") "
Nov 02 08:24:52.813557 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.813539    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-envoy\") pod \"1900eb00-365d-401d-a399-946fb0c03bfa\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") "
Nov 02 08:24:52.814104 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.812367    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1900eb00-365d-401d-a399-946fb0c03bfa-istiod-ca-cert" pod "1900eb00-365d-401d-a399-946fb0c03bfa" (UID: "1900eb00-365d-401d-a399-946fb0c03bfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:24:52.814104 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.813134    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-data" pod "1900eb00-365d-401d-a399-946fb0c03bfa" (UID: "1900eb00-365d-401d-a399-946fb0c03bfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:24:52.814358 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.814329    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-credential-socket" pod "1900eb00-365d-401d-a399-946fb0c03bfa" (UID: "1900eb00-365d-401d-a399-946fb0c03bfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:24:52.815419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.814482    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-istio-token\" (UniqueName: \"kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-istio-token\") pod \"1900eb00-365d-401d-a399-946fb0c03bfa\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") "
Nov 02 08:24:52.815419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.814512    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1900eb00-365d-401d-a399-946fb0c03bfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1900eb00-365d-401d-a399-946fb0c03bfa-istio-podinfo\") pod \"1900eb00-365d-401d-a399-946fb0c03bfa\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") "
Nov 02 08:24:52.815419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.814616    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-socket\") pod \"1900eb00-365d-401d-a399-946fb0c03bfa\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") "
Nov 02 08:24:52.815419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.814667    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-certs\") pod \"1900eb00-365d-401d-a399-946fb0c03bfa\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") "
Nov 02 08:24:52.815419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.814750    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-kube-api-access-l9r7v\" (UniqueName: \"kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-kube-api-access-l9r7v\") pod \"1900eb00-365d-401d-a399-946fb0c03bfa\" (UID: \"1900eb00-365d-401d-a399-946fb0c03bfa\") "
Nov 02 08:24:52.837912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.817261    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-envoy" pod "1900eb00-365d-401d-a399-946fb0c03bfa" (UID: "1900eb00-365d-401d-a399-946fb0c03bfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:24:52.837912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.826200    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-certs" pod "1900eb00-365d-401d-a399-946fb0c03bfa" (UID: "1900eb00-365d-401d-a399-946fb0c03bfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:24:52.837912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.830468    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-socket" pod "1900eb00-365d-401d-a399-946fb0c03bfa" (UID: "1900eb00-365d-401d-a399-946fb0c03bfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:24:52.837912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.831428    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1900eb00-365d-401d-a399-946fb0c03bfa-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:24:52.837912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.831458    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:24:52.837912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.831473    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:24:52.837912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.831489    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:24:52.837912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.831506    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:24:52.837912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.831521    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1900eb00-365d-401d-a399-946fb0c03bfa-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:24:52.842916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.842031    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1900eb00-365d-401d-a399-946fb0c03bfa-istio-podinfo" pod "1900eb00-365d-401d-a399-946fb0c03bfa" (UID: "1900eb00-365d-401d-a399-946fb0c03bfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:24:52.842916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.842826    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-istio-token" pod "1900eb00-365d-401d-a399-946fb0c03bfa" (UID: "1900eb00-365d-401d-a399-946fb0c03bfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:24:52.847001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.845129    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-kube-api-access-l9r7v" pod "1900eb00-365d-401d-a399-946fb0c03bfa" (UID: "1900eb00-365d-401d-a399-946fb0c03bfa"). InnerVolumeSpecName "kube-api-access-l9r7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:24:52.933766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.933730    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:24:52.933766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.933760    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1900eb00-365d-401d-a399-946fb0c03bfa-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:24:52.933766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:52.933772    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9r7v\" (UniqueName: \"kubernetes.io/projected/1900eb00-365d-401d-a399-946fb0c03bfa-kube-api-access-l9r7v\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:24:53.101318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.101213    4898 generic.go:356] "Generic (PLEG): container finished" podID="1900eb00-365d-401d-a399-946fb0c03bfa" containerID="eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1" exitCode=0
Nov 02 08:24:53.101318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.101261    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"
Nov 02 08:24:53.101318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.101293    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" event={"ID":"1900eb00-365d-401d-a399-946fb0c03bfa","Type":"ContainerDied","Data":"eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1"}
Nov 02 08:24:53.101631 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.101361    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf" event={"ID":"1900eb00-365d-401d-a399-946fb0c03bfa","Type":"ContainerDied","Data":"3f2a70fc1284aa29da0cab511f0aeedb8d9eddc334d911ddc201e4717c571547"}
Nov 02 08:24:53.101631 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.101392    4898 scope.go:122] "RemoveContainer" containerID="eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1"
Nov 02 08:24:53.114110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.114088    4898 scope.go:122] "RemoveContainer" containerID="f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c"
Nov 02 08:24:53.124715 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.124694    4898 scope.go:122] "RemoveContainer" containerID="eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1"
Nov 02 08:24:53.125028 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:24:53.125008    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1\": container with ID starting with eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1 not found: ID does not exist" containerID="eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1"
Nov 02 08:24:53.125076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.125039    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1"} err="failed to get container status \"eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1\": rpc error: code = NotFound desc = could not find container \"eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1\": container with ID starting with eff2a44675b15527a9098110d43c078fe66da2c0f9dd292a3ba3d4202dcf44b1 not found: ID does not exist"
Nov 02 08:24:53.125076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.125056    4898 scope.go:122] "RemoveContainer" containerID="f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c"
Nov 02 08:24:53.125334 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:24:53.125319    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c\": container with ID starting with f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c not found: ID does not exist" containerID="f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c"
Nov 02 08:24:53.125414 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.125337    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c"} err="failed to get container status \"f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c\": rpc error: code = NotFound desc = could not find container \"f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c\": container with ID starting with f88c3aac018d0ed98fd77f123127af5bfee88fe60af5ee12b308977e69e98d3c not found: ID does not exist"
Nov 02 08:24:53.164506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.164462    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"]
Nov 02 08:24:53.169771 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.169746    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dfd6f0-openshift-default-5685858dbf-ft9xf"]
Nov 02 08:24:53.469334 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:24:53.469295    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" path="/var/lib/kubelet/pods/1900eb00-365d-401d-a399-946fb0c03bfa/volumes"
Nov 02 08:25:14.993911 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:25:14.993846    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:25:25.276770 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.276728    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"]
Nov 02 08:25:25.317944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.303313    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy"
Nov 02 08:25:25.317944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.303346    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy"
Nov 02 08:25:25.317944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.303464    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0a3ec7ac-934c-4a5b-ab71-7e63615c34b7" containerName="revoke-keys"
Nov 02 08:25:25.317944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.303472    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a3ec7ac-934c-4a5b-ab71-7e63615c34b7" containerName="revoke-keys"
Nov 02 08:25:25.317944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.303653    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy"
Nov 02 08:25:25.317944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.303669    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a3ec7ac-934c-4a5b-ab71-7e63615c34b7" containerName="revoke-keys"
Nov 02 08:25:25.317944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.303684    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy"
Nov 02 08:25:25.319227 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.319189    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"]
Nov 02 08:25:25.319371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.319344    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:25.334815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.334766    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:25:25.346761 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.346736    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:25:25.346986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.346953    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:25:25.347063 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.346962    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:25:25.413770 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.413729    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125cda00-jv5jm\" (UID: \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:25.413961 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.413798    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xt9\" (UniqueName: \"kubernetes.io/projected/6c380a91-54c4-4d33-ba17-c5d82470bcaf-kube-api-access-g7xt9\") pod \"maas-api-revoke-sub-125cda00-jv5jm\" (UID: \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:25.414016 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.413994    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125cda00-jv5jm\" (UID: \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:25.515518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.515473    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125cda00-jv5jm\" (UID: \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:25.515727 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.515604    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xt9\" (UniqueName: \"kubernetes.io/projected/6c380a91-54c4-4d33-ba17-c5d82470bcaf-kube-api-access-g7xt9\") pod \"maas-api-revoke-sub-125cda00-jv5jm\" (UID: \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:25.515792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.515748    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125cda00-jv5jm\" (UID: \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:25.516693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.516665    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125cda00-jv5jm\" (UID: \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:25.518449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.518428    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125cda00-jv5jm\" (UID: \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:25.540061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.539980    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xt9\" (UniqueName: \"kubernetes.io/projected/6c380a91-54c4-4d33-ba17-c5d82470bcaf-kube-api-access-g7xt9\") pod \"maas-api-revoke-sub-125cda00-jv5jm\" (UID: \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:25.638026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.637991    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:25.961313 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:25.961277    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"]
Nov 02 08:25:25.961712 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:25:25.961687    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c380a91_54c4_4d33_ba17_c5d82470bcaf.slice/crio-90f150d7328e7931b2e26b44868cef27ed630957bc32beb34542aa6177f46ff2 WatchSource:0}: Error finding container 90f150d7328e7931b2e26b44868cef27ed630957bc32beb34542aa6177f46ff2: Status 404 returned error can't find the container with id 90f150d7328e7931b2e26b44868cef27ed630957bc32beb34542aa6177f46ff2
Nov 02 08:25:26.467580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.467541    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm" event={"ID":"6c380a91-54c4-4d33-ba17-c5d82470bcaf","Type":"ContainerStarted","Data":"90f150d7328e7931b2e26b44868cef27ed630957bc32beb34542aa6177f46ff2"}
Nov 02 08:25:26.768929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.768883    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"]
Nov 02 08:25:26.769760 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.769743    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy"
Nov 02 08:25:26.769811 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.769762    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1900eb00-365d-401d-a399-946fb0c03bfa" containerName="istio-proxy"
Nov 02 08:25:26.773673 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.773654    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:26.795217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.795182    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"]
Nov 02 08:25:26.836832 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.836788    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0208019-bbf4w\" (UID: \"99d2c733-9831-4afe-83d5-9df8920db621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:26.837140 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.837118    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0208019-bbf4w\" (UID: \"99d2c733-9831-4afe-83d5-9df8920db621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:26.837271 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.837243    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf28k\" (UniqueName: \"kubernetes.io/projected/99d2c733-9831-4afe-83d5-9df8920db621-kube-api-access-sf28k\") pod \"maas-api-revoke-sub-f0208019-bbf4w\" (UID: \"99d2c733-9831-4afe-83d5-9df8920db621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:26.938456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.938415    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0208019-bbf4w\" (UID: \"99d2c733-9831-4afe-83d5-9df8920db621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:26.938712 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.938683    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0208019-bbf4w\" (UID: \"99d2c733-9831-4afe-83d5-9df8920db621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:26.938813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.938769    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf28k\" (UniqueName: \"kubernetes.io/projected/99d2c733-9831-4afe-83d5-9df8920db621-kube-api-access-sf28k\") pod \"maas-api-revoke-sub-f0208019-bbf4w\" (UID: \"99d2c733-9831-4afe-83d5-9df8920db621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:26.939945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.939910    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0208019-bbf4w\" (UID: \"99d2c733-9831-4afe-83d5-9df8920db621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:26.941923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.941885    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0208019-bbf4w\" (UID: \"99d2c733-9831-4afe-83d5-9df8920db621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:26.964875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:26.964844    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf28k\" (UniqueName: \"kubernetes.io/projected/99d2c733-9831-4afe-83d5-9df8920db621-kube-api-access-sf28k\") pod \"maas-api-revoke-sub-f0208019-bbf4w\" (UID: \"99d2c733-9831-4afe-83d5-9df8920db621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:27.090956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:27.090567    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:27.435702 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:25:27.435666    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d2c733_9831_4afe_83d5_9df8920db621.slice/crio-879ff8d2b8f9be924bb2856790feffccb4c0a50577082be5b26b147304c57044 WatchSource:0}: Error finding container 879ff8d2b8f9be924bb2856790feffccb4c0a50577082be5b26b147304c57044: Status 404 returned error can't find the container with id 879ff8d2b8f9be924bb2856790feffccb4c0a50577082be5b26b147304c57044
Nov 02 08:25:27.436209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:27.436184    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"]
Nov 02 08:25:27.480447 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:27.480415    4898 generic.go:356] "Generic (PLEG): container finished" podID="6c380a91-54c4-4d33-ba17-c5d82470bcaf" containerID="2651bc06dec826050a74ba8d80e73546f3bfeb7d8301aa3576da0219468fcb2e" exitCode=0
Nov 02 08:25:27.480814 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:27.480498    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm" event={"ID":"6c380a91-54c4-4d33-ba17-c5d82470bcaf","Type":"ContainerDied","Data":"2651bc06dec826050a74ba8d80e73546f3bfeb7d8301aa3576da0219468fcb2e"}
Nov 02 08:25:27.482295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:27.482265    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w" event={"ID":"99d2c733-9831-4afe-83d5-9df8920db621","Type":"ContainerStarted","Data":"879ff8d2b8f9be924bb2856790feffccb4c0a50577082be5b26b147304c57044"}
Nov 02 08:25:28.497596 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:28.497557    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w" event={"ID":"99d2c733-9831-4afe-83d5-9df8920db621","Type":"ContainerStarted","Data":"3006b406edaccc710fc279a9ae804652c1819c634fda769cadf12dcc0a8b545d"}
Nov 02 08:25:28.540421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:28.540349    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w" podStartSLOduration=2.540329904 podStartE2EDuration="2.540329904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:25:28.539078246 +0000 UTC m=+3679.229405231" watchObservedRunningTime="2026-11-02 08:25:28.540329904 +0000 UTC m=+3679.230656870"
Nov 02 08:25:28.701025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:28.700988    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:28.794772 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:28.794671    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c380a91-54c4-4d33-ba17-c5d82470bcaf-kube-api-access-g7xt9\" (UniqueName: \"kubernetes.io/projected/6c380a91-54c4-4d33-ba17-c5d82470bcaf-kube-api-access-g7xt9\") pod \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\" (UID: \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\") "
Nov 02 08:25:28.794985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:28.794775    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-xks-ca\") pod \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\" (UID: \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\") "
Nov 02 08:25:28.794985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:28.794813    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-openshift-ca\") pod \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\" (UID: \"6c380a91-54c4-4d33-ba17-c5d82470bcaf\") "
Nov 02 08:25:28.796870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:28.796834    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-openshift-ca" pod "6c380a91-54c4-4d33-ba17-c5d82470bcaf" (UID: "6c380a91-54c4-4d33-ba17-c5d82470bcaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:25:28.802423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:28.802386    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-xks-ca" pod "6c380a91-54c4-4d33-ba17-c5d82470bcaf" (UID: "6c380a91-54c4-4d33-ba17-c5d82470bcaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:25:28.802616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:28.802426    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c380a91-54c4-4d33-ba17-c5d82470bcaf-kube-api-access-g7xt9" pod "6c380a91-54c4-4d33-ba17-c5d82470bcaf" (UID: "6c380a91-54c4-4d33-ba17-c5d82470bcaf"). InnerVolumeSpecName "kube-api-access-g7xt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:25:28.906913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:28.905319    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7xt9\" (UniqueName: \"kubernetes.io/projected/6c380a91-54c4-4d33-ba17-c5d82470bcaf-kube-api-access-g7xt9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:25:28.906913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:28.905362    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:25:28.906913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:28.905381    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c380a91-54c4-4d33-ba17-c5d82470bcaf-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:25:29.510498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:29.510461    4898 generic.go:356] "Generic (PLEG): container finished" podID="99d2c733-9831-4afe-83d5-9df8920db621" containerID="3006b406edaccc710fc279a9ae804652c1819c634fda769cadf12dcc0a8b545d" exitCode=0
Nov 02 08:25:29.511026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:29.510548    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w" event={"ID":"99d2c733-9831-4afe-83d5-9df8920db621","Type":"ContainerDied","Data":"3006b406edaccc710fc279a9ae804652c1819c634fda769cadf12dcc0a8b545d"}
Nov 02 08:25:29.512327 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:29.512301    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"
Nov 02 08:25:29.512477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:29.512336    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm" event={"ID":"6c380a91-54c4-4d33-ba17-c5d82470bcaf","Type":"ContainerDied","Data":"90f150d7328e7931b2e26b44868cef27ed630957bc32beb34542aa6177f46ff2"}
Nov 02 08:25:29.512477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:29.512382    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90f150d7328e7931b2e26b44868cef27ed630957bc32beb34542aa6177f46ff2"
Nov 02 08:25:30.824023 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:30.823993    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:30.938903 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:30.938862    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-xks-ca\") pod \"99d2c733-9831-4afe-83d5-9df8920db621\" (UID: \"99d2c733-9831-4afe-83d5-9df8920db621\") "
Nov 02 08:25:30.939073 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:30.939036    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d2c733-9831-4afe-83d5-9df8920db621-kube-api-access-sf28k\" (UniqueName: \"kubernetes.io/projected/99d2c733-9831-4afe-83d5-9df8920db621-kube-api-access-sf28k\") pod \"99d2c733-9831-4afe-83d5-9df8920db621\" (UID: \"99d2c733-9831-4afe-83d5-9df8920db621\") "
Nov 02 08:25:30.939145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:30.939116    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-openshift-ca\") pod \"99d2c733-9831-4afe-83d5-9df8920db621\" (UID: \"99d2c733-9831-4afe-83d5-9df8920db621\") "
Nov 02 08:25:30.939452 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:30.939425    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-openshift-ca" pod "99d2c733-9831-4afe-83d5-9df8920db621" (UID: "99d2c733-9831-4afe-83d5-9df8920db621"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:25:30.939836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:30.939818    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:25:30.941051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:30.941028    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-xks-ca" pod "99d2c733-9831-4afe-83d5-9df8920db621" (UID: "99d2c733-9831-4afe-83d5-9df8920db621"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:25:30.941206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:30.941185    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d2c733-9831-4afe-83d5-9df8920db621-kube-api-access-sf28k" pod "99d2c733-9831-4afe-83d5-9df8920db621" (UID: "99d2c733-9831-4afe-83d5-9df8920db621"). InnerVolumeSpecName "kube-api-access-sf28k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:25:31.040946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:31.040846    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d2c733-9831-4afe-83d5-9df8920db621-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:25:31.040946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:31.040876    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf28k\" (UniqueName: \"kubernetes.io/projected/99d2c733-9831-4afe-83d5-9df8920db621-kube-api-access-sf28k\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:25:31.534048 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:31.534018    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"
Nov 02 08:25:31.534222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:31.534052    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w" event={"ID":"99d2c733-9831-4afe-83d5-9df8920db621","Type":"ContainerDied","Data":"879ff8d2b8f9be924bb2856790feffccb4c0a50577082be5b26b147304c57044"}
Nov 02 08:25:31.534222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:31.534106    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="879ff8d2b8f9be924bb2856790feffccb4c0a50577082be5b26b147304c57044"
Nov 02 08:25:36.713738 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.713699    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"]
Nov 02 08:25:36.716323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.714497    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6c380a91-54c4-4d33-ba17-c5d82470bcaf" containerName="revoke-keys"
Nov 02 08:25:36.716323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.714513    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c380a91-54c4-4d33-ba17-c5d82470bcaf" containerName="revoke-keys"
Nov 02 08:25:36.716323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.714533    4898 cpu_manager.go:414] "containerMap: removing container" podUID="99d2c733-9831-4afe-83d5-9df8920db621" containerName="revoke-keys"
Nov 02 08:25:36.716323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.714540    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d2c733-9831-4afe-83d5-9df8920db621" containerName="revoke-keys"
Nov 02 08:25:36.716323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.714688    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d2c733-9831-4afe-83d5-9df8920db621" containerName="revoke-keys"
Nov 02 08:25:36.716323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.714712    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c380a91-54c4-4d33-ba17-c5d82470bcaf" containerName="revoke-keys"
Nov 02 08:25:36.718671 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.718654    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:36.739234 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.739203    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3859bc94-h5djk\" (UID: \"750d62da-25d2-4734-90ff-ce7976d45e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:36.739374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.739355    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3859bc94-h5djk\" (UID: \"750d62da-25d2-4734-90ff-ce7976d45e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:36.739442 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.739416    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z85gp\" (UniqueName: \"kubernetes.io/projected/750d62da-25d2-4734-90ff-ce7976d45e42-kube-api-access-z85gp\") pod \"maas-api-revoke-sub-3859bc94-h5djk\" (UID: \"750d62da-25d2-4734-90ff-ce7976d45e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:36.740203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.740176    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"]
Nov 02 08:25:36.746957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.744268    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:25:36.746957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.744529    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:25:36.746957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.744609    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:25:36.746957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.745936    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:25:36.840571 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.840529    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3859bc94-h5djk\" (UID: \"750d62da-25d2-4734-90ff-ce7976d45e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:36.840773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.840645    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85gp\" (UniqueName: \"kubernetes.io/projected/750d62da-25d2-4734-90ff-ce7976d45e42-kube-api-access-z85gp\") pod \"maas-api-revoke-sub-3859bc94-h5djk\" (UID: \"750d62da-25d2-4734-90ff-ce7976d45e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:36.840846 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.840767    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3859bc94-h5djk\" (UID: \"750d62da-25d2-4734-90ff-ce7976d45e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:36.841327 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.841307    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3859bc94-h5djk\" (UID: \"750d62da-25d2-4734-90ff-ce7976d45e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:36.843257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.843238    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3859bc94-h5djk\" (UID: \"750d62da-25d2-4734-90ff-ce7976d45e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:36.863833 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:36.863797    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85gp\" (UniqueName: \"kubernetes.io/projected/750d62da-25d2-4734-90ff-ce7976d45e42-kube-api-access-z85gp\") pod \"maas-api-revoke-sub-3859bc94-h5djk\" (UID: \"750d62da-25d2-4734-90ff-ce7976d45e42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:37.039475 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:37.039381    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:37.348761 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:25:37.348729    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750d62da_25d2_4734_90ff_ce7976d45e42.slice/crio-a08ee004db2366c6345c4ef0cb0bf674664a39d78ce446276308139e135b751d WatchSource:0}: Error finding container a08ee004db2366c6345c4ef0cb0bf674664a39d78ce446276308139e135b751d: Status 404 returned error can't find the container with id a08ee004db2366c6345c4ef0cb0bf674664a39d78ce446276308139e135b751d
Nov 02 08:25:37.349573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:37.349546    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"]
Nov 02 08:25:37.600136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:37.600056    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk" event={"ID":"750d62da-25d2-4734-90ff-ce7976d45e42","Type":"ContainerStarted","Data":"a08ee004db2366c6345c4ef0cb0bf674664a39d78ce446276308139e135b751d"}
Nov 02 08:25:38.613336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:38.613236    4898 generic.go:356] "Generic (PLEG): container finished" podID="750d62da-25d2-4734-90ff-ce7976d45e42" containerID="98c3263aa7ffea54d982adf9a762b576aaa3735ccb288295ae3dd19f79818a6b" exitCode=0
Nov 02 08:25:38.613750 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:38.613363    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk" event={"ID":"750d62da-25d2-4734-90ff-ce7976d45e42","Type":"ContainerDied","Data":"98c3263aa7ffea54d982adf9a762b576aaa3735ccb288295ae3dd19f79818a6b"}
Nov 02 08:25:39.886697 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:39.886630    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:40.017938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:40.017860    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750d62da-25d2-4734-90ff-ce7976d45e42-kube-api-access-z85gp\" (UniqueName: \"kubernetes.io/projected/750d62da-25d2-4734-90ff-ce7976d45e42-kube-api-access-z85gp\") pod \"750d62da-25d2-4734-90ff-ce7976d45e42\" (UID: \"750d62da-25d2-4734-90ff-ce7976d45e42\") "
Nov 02 08:25:40.018146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:40.018077    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-xks-ca\") pod \"750d62da-25d2-4734-90ff-ce7976d45e42\" (UID: \"750d62da-25d2-4734-90ff-ce7976d45e42\") "
Nov 02 08:25:40.018266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:40.018233    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-openshift-ca\") pod \"750d62da-25d2-4734-90ff-ce7976d45e42\" (UID: \"750d62da-25d2-4734-90ff-ce7976d45e42\") "
Nov 02 08:25:40.018640 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:40.018610    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-openshift-ca" pod "750d62da-25d2-4734-90ff-ce7976d45e42" (UID: "750d62da-25d2-4734-90ff-ce7976d45e42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:25:40.019201 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:40.019173    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:25:40.020421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:40.020395    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-xks-ca" pod "750d62da-25d2-4734-90ff-ce7976d45e42" (UID: "750d62da-25d2-4734-90ff-ce7976d45e42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:25:40.020707 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:40.020688    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750d62da-25d2-4734-90ff-ce7976d45e42-kube-api-access-z85gp" pod "750d62da-25d2-4734-90ff-ce7976d45e42" (UID: "750d62da-25d2-4734-90ff-ce7976d45e42"). InnerVolumeSpecName "kube-api-access-z85gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:25:40.120661 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:40.120622    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z85gp\" (UniqueName: \"kubernetes.io/projected/750d62da-25d2-4734-90ff-ce7976d45e42-kube-api-access-z85gp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:25:40.120661 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:40.120653    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750d62da-25d2-4734-90ff-ce7976d45e42-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:25:40.637685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:40.637651    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"
Nov 02 08:25:40.637860 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:40.637674    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk" event={"ID":"750d62da-25d2-4734-90ff-ce7976d45e42","Type":"ContainerDied","Data":"a08ee004db2366c6345c4ef0cb0bf674664a39d78ce446276308139e135b751d"}
Nov 02 08:25:40.637860 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:40.637738    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a08ee004db2366c6345c4ef0cb0bf674664a39d78ce446276308139e135b751d"
Nov 02 08:25:41.874926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:41.871357    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"]
Nov 02 08:25:41.874926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:41.872612    4898 cpu_manager.go:414] "containerMap: removing container" podUID="750d62da-25d2-4734-90ff-ce7976d45e42" containerName="revoke-keys"
Nov 02 08:25:41.874926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:41.872636    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750d62da-25d2-4734-90ff-ce7976d45e42" containerName="revoke-keys"
Nov 02 08:25:41.874926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:41.872863    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="750d62da-25d2-4734-90ff-ce7976d45e42" containerName="revoke-keys"
Nov 02 08:25:41.934422 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:41.934387    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"]
Nov 02 08:25:41.942943 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:41.934496    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:41.984284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:41.976172    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:25:41.984284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:41.976281    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:25:41.984284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:41.976315    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:25:41.984284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:41.976382    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:25:42.034823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:42.034790    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e574e612-gvv54\" (UID: \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:42.035114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:42.035095    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8xc5\" (UniqueName: \"kubernetes.io/projected/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-kube-api-access-f8xc5\") pod \"maas-api-revoke-sub-e574e612-gvv54\" (UID: \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:42.035221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:42.035211    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e574e612-gvv54\" (UID: \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:42.137043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:42.136939    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e574e612-gvv54\" (UID: \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:42.137239 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:42.137085    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8xc5\" (UniqueName: \"kubernetes.io/projected/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-kube-api-access-f8xc5\") pod \"maas-api-revoke-sub-e574e612-gvv54\" (UID: \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:42.137239 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:42.137144    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e574e612-gvv54\" (UID: \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:42.137812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:42.137790    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e574e612-gvv54\" (UID: \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:42.139451 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:42.139424    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e574e612-gvv54\" (UID: \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:42.164664 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:42.164630    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8xc5\" (UniqueName: \"kubernetes.io/projected/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-kube-api-access-f8xc5\") pod \"maas-api-revoke-sub-e574e612-gvv54\" (UID: \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:42.272857 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:42.272586    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:42.961992 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:42.961952    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"]
Nov 02 08:25:43.810919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:43.810853    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54" event={"ID":"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1","Type":"ContainerStarted","Data":"737e70147b57f6c9aa7c5673ee2bfd36574ffca4bedc7cab48d4cfeae0ae504b"}
Nov 02 08:25:43.811283 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:43.811258    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54" event={"ID":"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1","Type":"ContainerStarted","Data":"936afcc937e21428db206072573547da33072871249ec94190ac2ec8e558d3fa"}
Nov 02 08:25:43.855300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:43.855243    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54" podStartSLOduration=2.855229037 podStartE2EDuration="2.855229037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:25:43.851722637 +0000 UTC m=+3694.542049594" watchObservedRunningTime="2026-11-02 08:25:43.855229037 +0000 UTC m=+3694.545556001"
Nov 02 08:25:44.820775 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:44.820739    4898 generic.go:356] "Generic (PLEG): container finished" podID="04cdcd7c-7bd5-48e6-885e-29d3b8049bf1" containerID="737e70147b57f6c9aa7c5673ee2bfd36574ffca4bedc7cab48d4cfeae0ae504b" exitCode=0
Nov 02 08:25:44.821234 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:44.820832    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54" event={"ID":"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1","Type":"ContainerDied","Data":"737e70147b57f6c9aa7c5673ee2bfd36574ffca4bedc7cab48d4cfeae0ae504b"}
Nov 02 08:25:44.989581 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:25:44.989549    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:25:46.020870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.020841    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:46.133298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.133206    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-kube-api-access-f8xc5\" (UniqueName: \"kubernetes.io/projected/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-kube-api-access-f8xc5\") pod \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\" (UID: \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\") "
Nov 02 08:25:46.133709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.133600    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-xks-ca\") pod \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\" (UID: \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\") "
Nov 02 08:25:46.134257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.134217    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-openshift-ca" pod "04cdcd7c-7bd5-48e6-885e-29d3b8049bf1" (UID: "04cdcd7c-7bd5-48e6-885e-29d3b8049bf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:25:46.134412 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.134290    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-openshift-ca\") pod \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\" (UID: \"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1\") "
Nov 02 08:25:46.135080 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.135052    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:25:46.135777 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.135745    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-kube-api-access-f8xc5" pod "04cdcd7c-7bd5-48e6-885e-29d3b8049bf1" (UID: "04cdcd7c-7bd5-48e6-885e-29d3b8049bf1"). InnerVolumeSpecName "kube-api-access-f8xc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:25:46.135916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.135800    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-xks-ca" pod "04cdcd7c-7bd5-48e6-885e-29d3b8049bf1" (UID: "04cdcd7c-7bd5-48e6-885e-29d3b8049bf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:25:46.236522 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.236486    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:25:46.236522 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.236518    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8xc5\" (UniqueName: \"kubernetes.io/projected/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1-kube-api-access-f8xc5\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:25:46.842391 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.842350    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54" event={"ID":"04cdcd7c-7bd5-48e6-885e-29d3b8049bf1","Type":"ContainerDied","Data":"936afcc937e21428db206072573547da33072871249ec94190ac2ec8e558d3fa"}
Nov 02 08:25:46.842391 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.842387    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"
Nov 02 08:25:46.842610 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:46.842412    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="936afcc937e21428db206072573547da33072871249ec94190ac2ec8e558d3fa"
Nov 02 08:25:59.241836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:25:59.241800    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"]
Nov 02 08:26:07.428489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.428449    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"]
Nov 02 08:26:07.429333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.429314    4898 cpu_manager.go:414] "containerMap: removing container" podUID="04cdcd7c-7bd5-48e6-885e-29d3b8049bf1" containerName="revoke-keys"
Nov 02 08:26:07.429333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.429334    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04cdcd7c-7bd5-48e6-885e-29d3b8049bf1" containerName="revoke-keys"
Nov 02 08:26:07.429498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.429488    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="04cdcd7c-7bd5-48e6-885e-29d3b8049bf1" containerName="revoke-keys"
Nov 02 08:26:07.434668 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.434644    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:07.456027 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.456001    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:26:07.456298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.456280    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:26:07.463877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.463853    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:26:07.464070 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.463980    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:26:07.474372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.474340    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"]
Nov 02 08:26:07.480169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.480120    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4551b1b-251b-4e79-b202-895272200c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9006596f-9c5wq\" (UID: \"a4551b1b-251b-4e79-b202-895272200c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:07.480291 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.480269    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkplg\" (UniqueName: \"kubernetes.io/projected/a4551b1b-251b-4e79-b202-895272200c23-kube-api-access-tkplg\") pod \"maas-api-revoke-sub-9006596f-9c5wq\" (UID: \"a4551b1b-251b-4e79-b202-895272200c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:07.480398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.480383    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4551b1b-251b-4e79-b202-895272200c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9006596f-9c5wq\" (UID: \"a4551b1b-251b-4e79-b202-895272200c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:07.582727 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.582681    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4551b1b-251b-4e79-b202-895272200c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9006596f-9c5wq\" (UID: \"a4551b1b-251b-4e79-b202-895272200c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:07.582948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.582845    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkplg\" (UniqueName: \"kubernetes.io/projected/a4551b1b-251b-4e79-b202-895272200c23-kube-api-access-tkplg\") pod \"maas-api-revoke-sub-9006596f-9c5wq\" (UID: \"a4551b1b-251b-4e79-b202-895272200c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:07.585918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.583031    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4551b1b-251b-4e79-b202-895272200c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9006596f-9c5wq\" (UID: \"a4551b1b-251b-4e79-b202-895272200c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:07.585918 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.584033    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4551b1b-251b-4e79-b202-895272200c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9006596f-9c5wq\" (UID: \"a4551b1b-251b-4e79-b202-895272200c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:07.591359 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.591327    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4551b1b-251b-4e79-b202-895272200c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9006596f-9c5wq\" (UID: \"a4551b1b-251b-4e79-b202-895272200c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:07.609396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.609372    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkplg\" (UniqueName: \"kubernetes.io/projected/a4551b1b-251b-4e79-b202-895272200c23-kube-api-access-tkplg\") pod \"maas-api-revoke-sub-9006596f-9c5wq\" (UID: \"a4551b1b-251b-4e79-b202-895272200c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:07.752624 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:07.752539    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:08.124467 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:26:08.124430    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4551b1b_251b_4e79_b202_895272200c23.slice/crio-5725c3bf4f70a96bc400b37d7a5b3dd943e6888cd928e4c4e7b0cc7dbf2fdd2a WatchSource:0}: Error finding container 5725c3bf4f70a96bc400b37d7a5b3dd943e6888cd928e4c4e7b0cc7dbf2fdd2a: Status 404 returned error can't find the container with id 5725c3bf4f70a96bc400b37d7a5b3dd943e6888cd928e4c4e7b0cc7dbf2fdd2a
Nov 02 08:26:08.125620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:08.125591    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"]
Nov 02 08:26:08.144610 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:08.144580    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq" event={"ID":"a4551b1b-251b-4e79-b202-895272200c23","Type":"ContainerStarted","Data":"5725c3bf4f70a96bc400b37d7a5b3dd943e6888cd928e4c4e7b0cc7dbf2fdd2a"}
Nov 02 08:26:09.155796 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:09.155715    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq" event={"ID":"a4551b1b-251b-4e79-b202-895272200c23","Type":"ContainerStarted","Data":"4c0bfff422b788fa70123084981cc58c62691dac218cafd6657800aa5bfe0a40"}
Nov 02 08:26:09.196686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:09.196641    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq" podStartSLOduration=2.196627227 podStartE2EDuration="2.196627227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:26:09.195434838 +0000 UTC m=+3719.885761832" watchObservedRunningTime="2026-11-02 08:26:09.196627227 +0000 UTC m=+3719.886954191"
Nov 02 08:26:10.168436 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:10.168396    4898 generic.go:356] "Generic (PLEG): container finished" podID="a4551b1b-251b-4e79-b202-895272200c23" containerID="4c0bfff422b788fa70123084981cc58c62691dac218cafd6657800aa5bfe0a40" exitCode=0
Nov 02 08:26:10.168950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:10.168485    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq" event={"ID":"a4551b1b-251b-4e79-b202-895272200c23","Type":"ContainerDied","Data":"4c0bfff422b788fa70123084981cc58c62691dac218cafd6657800aa5bfe0a40"}
Nov 02 08:26:11.366414 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:11.366382    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:11.457677 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:11.457593    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4551b1b-251b-4e79-b202-895272200c23-kube-api-access-tkplg\" (UniqueName: \"kubernetes.io/projected/a4551b1b-251b-4e79-b202-895272200c23-kube-api-access-tkplg\") pod \"a4551b1b-251b-4e79-b202-895272200c23\" (UID: \"a4551b1b-251b-4e79-b202-895272200c23\") "
Nov 02 08:26:11.458223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:11.458194    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4551b1b-251b-4e79-b202-895272200c23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4551b1b-251b-4e79-b202-895272200c23-maas-api-openshift-ca\") pod \"a4551b1b-251b-4e79-b202-895272200c23\" (UID: \"a4551b1b-251b-4e79-b202-895272200c23\") "
Nov 02 08:26:11.458466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:11.458444    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4551b1b-251b-4e79-b202-895272200c23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4551b1b-251b-4e79-b202-895272200c23-maas-api-xks-ca\") pod \"a4551b1b-251b-4e79-b202-895272200c23\" (UID: \"a4551b1b-251b-4e79-b202-895272200c23\") "
Nov 02 08:26:11.458995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:11.458879    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4551b1b-251b-4e79-b202-895272200c23-maas-api-openshift-ca" pod "a4551b1b-251b-4e79-b202-895272200c23" (UID: "a4551b1b-251b-4e79-b202-895272200c23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:26:11.459320 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:11.459297    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4551b1b-251b-4e79-b202-895272200c23-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:11.462073 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:11.462047    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4551b1b-251b-4e79-b202-895272200c23-kube-api-access-tkplg" pod "a4551b1b-251b-4e79-b202-895272200c23" (UID: "a4551b1b-251b-4e79-b202-895272200c23"). InnerVolumeSpecName "kube-api-access-tkplg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:11.462298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:11.462276    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4551b1b-251b-4e79-b202-895272200c23-maas-api-xks-ca" pod "a4551b1b-251b-4e79-b202-895272200c23" (UID: "a4551b1b-251b-4e79-b202-895272200c23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:26:11.561025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:11.560990    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4551b1b-251b-4e79-b202-895272200c23-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:11.561284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:11.561255    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkplg\" (UniqueName: \"kubernetes.io/projected/a4551b1b-251b-4e79-b202-895272200c23-kube-api-access-tkplg\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:12.190453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:12.190418    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"
Nov 02 08:26:12.190453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:12.190441    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq" event={"ID":"a4551b1b-251b-4e79-b202-895272200c23","Type":"ContainerDied","Data":"5725c3bf4f70a96bc400b37d7a5b3dd943e6888cd928e4c4e7b0cc7dbf2fdd2a"}
Nov 02 08:26:12.190686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:12.190501    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5725c3bf4f70a96bc400b37d7a5b3dd943e6888cd928e4c4e7b0cc7dbf2fdd2a"
Nov 02 08:26:14.272710 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.272673    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp" podUID="cc741a4f-4c12-41e4-98e7-6a9231c6f00e" containerName="main" containerID="cri-o://43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b" gracePeriod=45
Nov 02 08:26:14.600271 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.600232    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:26:14.726694 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.726660    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-model-cache\") pod \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") "
Nov 02 08:26:14.726872 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.726701    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tmp-dir\") pod \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") "
Nov 02 08:26:14.726872 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.726751    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-home\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-home\") pod \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") "
Nov 02 08:26:14.726872 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.726863    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tls-certs\") pod \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") "
Nov 02 08:26:14.727066 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.726931    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-dshm\") pod \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") "
Nov 02 08:26:14.727066 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.726990    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-model-cache" pod "cc741a4f-4c12-41e4-98e7-6a9231c6f00e" (UID: "cc741a4f-4c12-41e4-98e7-6a9231c6f00e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:14.727133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.727055    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tmp-dir" pod "cc741a4f-4c12-41e4-98e7-6a9231c6f00e" (UID: "cc741a4f-4c12-41e4-98e7-6a9231c6f00e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:14.727133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.727092    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-home" pod "cc741a4f-4c12-41e4-98e7-6a9231c6f00e" (UID: "cc741a4f-4c12-41e4-98e7-6a9231c6f00e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:14.727208 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.727128    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-kube-api-access-pbgvv\" (UniqueName: \"kubernetes.io/projected/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-kube-api-access-pbgvv\") pod \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\" (UID: \"cc741a4f-4c12-41e4-98e7-6a9231c6f00e\") "
Nov 02 08:26:14.727780 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.727752    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:14.727780 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.727773    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:14.727780 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.727781    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:14.729197 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.729173    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-dshm" pod "cc741a4f-4c12-41e4-98e7-6a9231c6f00e" (UID: "cc741a4f-4c12-41e4-98e7-6a9231c6f00e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:14.729320 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.729234    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tls-certs" pod "cc741a4f-4c12-41e4-98e7-6a9231c6f00e" (UID: "cc741a4f-4c12-41e4-98e7-6a9231c6f00e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:26:14.729320 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.729306    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-kube-api-access-pbgvv" pod "cc741a4f-4c12-41e4-98e7-6a9231c6f00e" (UID: "cc741a4f-4c12-41e4-98e7-6a9231c6f00e"). InnerVolumeSpecName "kube-api-access-pbgvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:14.829445 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.829361    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:14.829445 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.829393    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:14.829445 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:14.829407    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbgvv\" (UniqueName: \"kubernetes.io/projected/cc741a4f-4c12-41e4-98e7-6a9231c6f00e-kube-api-access-pbgvv\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:14.990103 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:26:14.990052    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:26:15.222864 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:15.222831    4898 generic.go:356] "Generic (PLEG): container finished" podID="cc741a4f-4c12-41e4-98e7-6a9231c6f00e" containerID="43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b" exitCode=0
Nov 02 08:26:15.223106 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:15.222924    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp" event={"ID":"cc741a4f-4c12-41e4-98e7-6a9231c6f00e","Type":"ContainerDied","Data":"43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b"}
Nov 02 08:26:15.223106 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:15.222951    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"
Nov 02 08:26:15.223106 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:15.222967    4898 scope.go:122] "RemoveContainer" containerID="43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b"
Nov 02 08:26:15.223106 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:15.222957    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp" event={"ID":"cc741a4f-4c12-41e4-98e7-6a9231c6f00e","Type":"ContainerDied","Data":"78aefa919b67993e112b93e75e42a199ba952d618de831c325603c845f9aac3f"}
Nov 02 08:26:15.235576 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:15.235550    4898 scope.go:122] "RemoveContainer" containerID="43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b"
Nov 02 08:26:15.235852 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:26:15.235830    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b\": container with ID starting with 43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b not found: ID does not exist" containerID="43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b"
Nov 02 08:26:15.235982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:15.235860    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b"} err="failed to get container status \"43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b\": rpc error: code = NotFound desc = could not find container \"43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b\": container with ID starting with 43328766a5e36502563c074be39c1e01018f63864598bbbfce2918f25b2db51b not found: ID does not exist"
Nov 02 08:26:15.279328 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:15.279295    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"]
Nov 02 08:26:15.284043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:15.284016    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ba44ed/test-model-ba44ed-kserve-659c6df84c-spdfp"]
Nov 02 08:26:15.468304 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:15.468263    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc741a4f-4c12-41e4-98e7-6a9231c6f00e" path="/var/lib/kubelet/pods/cc741a4f-4c12-41e4-98e7-6a9231c6f00e/volumes"
Nov 02 08:26:21.169232 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.169192    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"]
Nov 02 08:26:21.169981 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.169963    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a4551b1b-251b-4e79-b202-895272200c23" containerName="revoke-keys"
Nov 02 08:26:21.169981 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.169981    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4551b1b-251b-4e79-b202-895272200c23" containerName="revoke-keys"
Nov 02 08:26:21.170121 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.170014    4898 cpu_manager.go:414] "containerMap: removing container" podUID="cc741a4f-4c12-41e4-98e7-6a9231c6f00e" containerName="main"
Nov 02 08:26:21.170121 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.170020    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc741a4f-4c12-41e4-98e7-6a9231c6f00e" containerName="main"
Nov 02 08:26:21.170236 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.170138    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4551b1b-251b-4e79-b202-895272200c23" containerName="revoke-keys"
Nov 02 08:26:21.170236 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.170147    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc741a4f-4c12-41e4-98e7-6a9231c6f00e" containerName="main"
Nov 02 08:26:21.174090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.174070    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:21.195327 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.195299    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"]
Nov 02 08:26:21.200042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.200016    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:26:21.200336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.200316    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:26:21.200504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.200489    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:26:21.202140 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.202111    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:26:21.348410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.348335    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdgx\" (UniqueName: \"kubernetes.io/projected/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-kube-api-access-4jdgx\") pod \"maas-api-revoke-sub-cff60616-99rwd\" (UID: \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:21.348604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.348507    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff60616-99rwd\" (UID: \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:21.348660 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.348644    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff60616-99rwd\" (UID: \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:21.449990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.449878    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff60616-99rwd\" (UID: \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:21.450182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.449993    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff60616-99rwd\" (UID: \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:21.450182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.450118    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdgx\" (UniqueName: \"kubernetes.io/projected/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-kube-api-access-4jdgx\") pod \"maas-api-revoke-sub-cff60616-99rwd\" (UID: \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:21.450588 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.450561    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff60616-99rwd\" (UID: \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:21.452392 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.452367    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff60616-99rwd\" (UID: \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:21.483705 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.483665    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdgx\" (UniqueName: \"kubernetes.io/projected/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-kube-api-access-4jdgx\") pod \"maas-api-revoke-sub-cff60616-99rwd\" (UID: \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:21.490038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.490012    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:21.827213 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:26:21.827170    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8b0a1d_1a47_4014_8f56_e13d9cd4bb43.slice/crio-f76d537b9c343215cace571550522c5b3543a290bb1a4fee32ecd50ce1c04561 WatchSource:0}: Error finding container f76d537b9c343215cace571550522c5b3543a290bb1a4fee32ecd50ce1c04561: Status 404 returned error can't find the container with id f76d537b9c343215cace571550522c5b3543a290bb1a4fee32ecd50ce1c04561
Nov 02 08:26:21.827964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:21.827860    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"]
Nov 02 08:26:22.288827 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:22.288793    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd" event={"ID":"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43","Type":"ContainerStarted","Data":"e4c02f7cb4ee1430b96b723ddd7f53192ed148b2b23b57f48ed493265381d6e2"}
Nov 02 08:26:22.289239 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:22.288859    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd" event={"ID":"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43","Type":"ContainerStarted","Data":"f76d537b9c343215cace571550522c5b3543a290bb1a4fee32ecd50ce1c04561"}
Nov 02 08:26:22.335419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:22.335374    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd" podStartSLOduration=1.33535979 podStartE2EDuration="1.33535979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:26:22.331023281 +0000 UTC m=+3733.021350258" watchObservedRunningTime="2026-11-02 08:26:22.33535979 +0000 UTC m=+3733.025686803"
Nov 02 08:26:23.299155 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:23.299113    4898 generic.go:356] "Generic (PLEG): container finished" podID="ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43" containerID="e4c02f7cb4ee1430b96b723ddd7f53192ed148b2b23b57f48ed493265381d6e2" exitCode=0
Nov 02 08:26:23.299661 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:23.299213    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd" event={"ID":"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43","Type":"ContainerDied","Data":"e4c02f7cb4ee1430b96b723ddd7f53192ed148b2b23b57f48ed493265381d6e2"}
Nov 02 08:26:24.655198 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:24.655166    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:24.849331 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:24.849251    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-xks-ca\") pod \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\" (UID: \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\") "
Nov 02 08:26:24.849489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:24.849387    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-openshift-ca\") pod \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\" (UID: \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\") "
Nov 02 08:26:24.849489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:24.849441    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-kube-api-access-4jdgx\" (UniqueName: \"kubernetes.io/projected/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-kube-api-access-4jdgx\") pod \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\" (UID: \"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43\") "
Nov 02 08:26:24.849739 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:24.849708    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-openshift-ca" pod "ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43" (UID: "ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:26:24.850069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:24.850049    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:24.851561 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:24.851522    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-xks-ca" pod "ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43" (UID: "ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:26:24.851657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:24.851640    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-kube-api-access-4jdgx" pod "ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43" (UID: "ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43"). InnerVolumeSpecName "kube-api-access-4jdgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:24.951371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:24.951341    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdgx\" (UniqueName: \"kubernetes.io/projected/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-kube-api-access-4jdgx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:24.951371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:24.951369    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:25.419406 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:25.419371    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"
Nov 02 08:26:25.419606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:25.419376    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd" event={"ID":"ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43","Type":"ContainerDied","Data":"f76d537b9c343215cace571550522c5b3543a290bb1a4fee32ecd50ce1c04561"}
Nov 02 08:26:25.419606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:25.419513    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f76d537b9c343215cace571550522c5b3543a290bb1a4fee32ecd50ce1c04561"
Nov 02 08:26:37.540149 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:37.540112    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"]
Nov 02 08:26:40.173929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:40.172500    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"]
Nov 02 08:26:40.173929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:40.172791    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy" containerID="cri-o://19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d" gracePeriod=30
Nov 02 08:26:42.541472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:42.541400    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb" podUID="26618807-5b73-42bf-ad76-b3e5c05a9055" containerName="payload-pre-processing" containerID="cri-o://71627ac25cd2e36f41b838aaf7d46fffc9b95293ff744c95cfb520685002673a" gracePeriod=25
Nov 02 08:26:44.991153 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:26:44.991108    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:26:45.582541 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.582512    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:26:45.630846 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.630814    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-credential-socket\") pod \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") "
Nov 02 08:26:45.631046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.630881    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-envoy\") pod \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") "
Nov 02 08:26:45.631046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.630979    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-token\" (UniqueName: \"kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-token\") pod \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") "
Nov 02 08:26:45.631181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.631101    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-podinfo\") pod \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") "
Nov 02 08:26:45.631181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.631130    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-socket\") pod \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") "
Nov 02 08:26:45.631318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.631173    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-credential-socket" pod "ed94547d-f04f-4401-a356-fd38ffdcf2e4" (UID: "ed94547d-f04f-4401-a356-fd38ffdcf2e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:45.631318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.631209    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-certs\") pod \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") "
Nov 02 08:26:45.631318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.631268    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istiod-ca-cert\") pod \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") "
Nov 02 08:26:45.632162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.631336    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-kube-api-access-d6v6k\" (UniqueName: \"kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-kube-api-access-d6v6k\") pod \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") "
Nov 02 08:26:45.632162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.631389    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-data\") pod \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\" (UID: \"ed94547d-f04f-4401-a356-fd38ffdcf2e4\") "
Nov 02 08:26:45.632162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.631561    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-certs" pod "ed94547d-f04f-4401-a356-fd38ffdcf2e4" (UID: "ed94547d-f04f-4401-a356-fd38ffdcf2e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:45.632162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.631733    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-socket" pod "ed94547d-f04f-4401-a356-fd38ffdcf2e4" (UID: "ed94547d-f04f-4401-a356-fd38ffdcf2e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:45.632162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.631937    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-data" pod "ed94547d-f04f-4401-a356-fd38ffdcf2e4" (UID: "ed94547d-f04f-4401-a356-fd38ffdcf2e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:45.632162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.632100    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istiod-ca-cert" pod "ed94547d-f04f-4401-a356-fd38ffdcf2e4" (UID: "ed94547d-f04f-4401-a356-fd38ffdcf2e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:26:45.632426 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.632308    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:45.632426 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.632329    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:45.632426 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.632343    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:45.632426 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.632359    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:45.632426 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.632373    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:45.633525 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.633493    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-podinfo" pod "ed94547d-f04f-4401-a356-fd38ffdcf2e4" (UID: "ed94547d-f04f-4401-a356-fd38ffdcf2e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:26:45.633658 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.633619    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-token" pod "ed94547d-f04f-4401-a356-fd38ffdcf2e4" (UID: "ed94547d-f04f-4401-a356-fd38ffdcf2e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:45.634199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.634176    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-kube-api-access-d6v6k" pod "ed94547d-f04f-4401-a356-fd38ffdcf2e4" (UID: "ed94547d-f04f-4401-a356-fd38ffdcf2e4"). InnerVolumeSpecName "kube-api-access-d6v6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:45.634299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.634254    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-envoy" pod "ed94547d-f04f-4401-a356-fd38ffdcf2e4" (UID: "ed94547d-f04f-4401-a356-fd38ffdcf2e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:26:45.704817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.704782    4898 generic.go:356] "Generic (PLEG): container finished" podID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerID="19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d" exitCode=0
Nov 02 08:26:45.705017 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.704836    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"
Nov 02 08:26:45.705017 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.704862    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" event={"ID":"ed94547d-f04f-4401-a356-fd38ffdcf2e4","Type":"ContainerDied","Data":"19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d"}
Nov 02 08:26:45.705017 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.704958    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q" event={"ID":"ed94547d-f04f-4401-a356-fd38ffdcf2e4","Type":"ContainerDied","Data":"6d8445317b9af1c5b65af816fe65ef1ed65ae70ebbfe8d8ad9ff6fdc5a56a6eb"}
Nov 02 08:26:45.705017 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.704993    4898 scope.go:122] "RemoveContainer" containerID="19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d"
Nov 02 08:26:45.719551 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.719532    4898 scope.go:122] "RemoveContainer" containerID="4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37"
Nov 02 08:26:45.731340 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.731313    4898 scope.go:122] "RemoveContainer" containerID="19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d"
Nov 02 08:26:45.731680 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:26:45.731651    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d\": container with ID starting with 19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d not found: ID does not exist" containerID="19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d"
Nov 02 08:26:45.731765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.731690    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d"} err="failed to get container status \"19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d\": rpc error: code = NotFound desc = could not find container \"19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d\": container with ID starting with 19aa804a8870ce135bb8d485599c561826d1b5f31d914d6fd1cf9672e1e0cc0d not found: ID does not exist"
Nov 02 08:26:45.731765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.731717    4898 scope.go:122] "RemoveContainer" containerID="4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37"
Nov 02 08:26:45.732080 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:26:45.732049    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37\": container with ID starting with 4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37 not found: ID does not exist" containerID="4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37"
Nov 02 08:26:45.732217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.732084    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37"} err="failed to get container status \"4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37\": rpc error: code = NotFound desc = could not find container \"4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37\": container with ID starting with 4ccdefa6953201684ac6e9fdbde8f00f1b8ecb76c89efe895c2e256b5be44e37 not found: ID does not exist"
Nov 02 08:26:45.734158 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.734135    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6v6k\" (UniqueName: \"kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-kube-api-access-d6v6k\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:45.734158 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.734158    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:45.734329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.734168    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:45.734329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.734177    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed94547d-f04f-4401-a356-fd38ffdcf2e4-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:45.810205 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.810137    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"]
Nov 02 08:26:45.815031 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:45.815008    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e6f4fe-openshift-default-6cfbbc7bb7-rlv7q"]
Nov 02 08:26:47.034836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:47.034796    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"]
Nov 02 08:26:47.046491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:47.046460    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d906200-dvmpb"]
Nov 02 08:26:47.469097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:47.469059    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36fb71c6-d4e0-408f-bdd1-0c6ce85a9156" path="/var/lib/kubelet/pods/36fb71c6-d4e0-408f-bdd1-0c6ce85a9156/volumes"
Nov 02 08:26:47.469841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:47.469821    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" path="/var/lib/kubelet/pods/ed94547d-f04f-4401-a356-fd38ffdcf2e4/volumes"
Nov 02 08:26:48.150090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:48.150053    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"]
Nov 02 08:26:48.179480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:48.179447    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"]
Nov 02 08:26:52.007128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:52.007086    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb" podUID="26618807-5b73-42bf-ad76-b3e5c05a9055" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:26:53.151354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.151285    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht" podUID="2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8" containerName="payload-processing" containerID="cri-o://a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8" gracePeriod=25
Nov 02 08:26:53.179876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.179838    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z" podUID="c640c1dc-baae-4bb0-acdb-a5b09e732e55" containerName="payload-pre-processing" containerID="cri-o://1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882" gracePeriod=25
Nov 02 08:26:53.454353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.454324    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"
Nov 02 08:26:53.591037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.590997    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-plugins-config-volume\") pod \"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8\" (UID: \"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8\") "
Nov 02 08:26:53.591262 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.591246    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-kube-api-access-8nrjz\" (UniqueName: \"kubernetes.io/projected/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-kube-api-access-8nrjz\") pod \"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8\" (UID: \"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8\") "
Nov 02 08:26:53.591416 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.591388    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-plugins-config-volume" pod "2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8" (UID: "2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:26:53.592041 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.592010    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:53.593552 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.593518    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-kube-api-access-8nrjz" pod "2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8" (UID: "2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8"). InnerVolumeSpecName "kube-api-access-8nrjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:26:53.693823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.693757    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nrjz\" (UniqueName: \"kubernetes.io/projected/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8-kube-api-access-8nrjz\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:26:53.869608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.869563    4898 generic.go:356] "Generic (PLEG): container finished" podID="2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8" containerID="a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8" exitCode=0
Nov 02 08:26:53.869811 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.869697    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht" event={"ID":"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8","Type":"ContainerDied","Data":"a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8"}
Nov 02 08:26:53.869811 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.869794    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht" event={"ID":"2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8","Type":"ContainerDied","Data":"8a8ded5c668aac309699554974e629986c47e383997043af07c18db2ef24d211"}
Nov 02 08:26:53.870015 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.869828    4898 scope.go:122] "RemoveContainer" containerID="a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8"
Nov 02 08:26:53.870090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.870073    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"
Nov 02 08:26:53.897267 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.897241    4898 scope.go:122] "RemoveContainer" containerID="a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8"
Nov 02 08:26:53.897567 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:26:53.897545    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8\": container with ID starting with a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8 not found: ID does not exist" containerID="a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8"
Nov 02 08:26:53.897681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.897576    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8"} err="failed to get container status \"a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8\": rpc error: code = NotFound desc = could not find container \"a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8\": container with ID starting with a5f037f9be0299d2311f5aafd00c63b28ded4b95aa003a4dc74a1db80161a0c8 not found: ID does not exist"
Nov 02 08:26:53.927843 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.927813    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"]
Nov 02 08:26:53.932507 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:53.932479    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ba44ed-5d5db5f6ff-pbpht"]
Nov 02 08:26:55.467500 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:55.467464    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8" path="/var/lib/kubelet/pods/2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8/volumes"
Nov 02 08:26:58.032384 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:58.032348    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"]
Nov 02 08:26:58.044279 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:58.044248    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f3e839-5sb6l"]
Nov 02 08:26:59.209433 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:59.209392    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z" podUID="c640c1dc-baae-4bb0-acdb-a5b09e732e55" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:26:59.467985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:26:59.467870    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed378475-ec00-4728-8580-8fc7cd8a57bc" path="/var/lib/kubelet/pods/ed378475-ec00-4728-8580-8fc7cd8a57bc/volumes"
Nov 02 08:27:02.007070 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:02.007027    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb" podUID="26618807-5b73-42bf-ad76-b3e5c05a9055" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:27:03.038272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:03.038241    4898 generic.go:356] "Generic (PLEG): container finished" podID="26618807-5b73-42bf-ad76-b3e5c05a9055" containerID="71627ac25cd2e36f41b838aaf7d46fffc9b95293ff744c95cfb520685002673a" exitCode=0
Nov 02 08:27:03.038931 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:03.038906    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb" event={"ID":"26618807-5b73-42bf-ad76-b3e5c05a9055","Type":"ContainerDied","Data":"71627ac25cd2e36f41b838aaf7d46fffc9b95293ff744c95cfb520685002673a"}
Nov 02 08:27:03.057446 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:03.057423    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"
Nov 02 08:27:03.210145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:03.210105    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26618807-5b73-42bf-ad76-b3e5c05a9055-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26618807-5b73-42bf-ad76-b3e5c05a9055-plugins-config-volume\") pod \"26618807-5b73-42bf-ad76-b3e5c05a9055\" (UID: \"26618807-5b73-42bf-ad76-b3e5c05a9055\") "
Nov 02 08:27:03.210349 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:03.210252    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26618807-5b73-42bf-ad76-b3e5c05a9055-kube-api-access-v7j74\" (UniqueName: \"kubernetes.io/projected/26618807-5b73-42bf-ad76-b3e5c05a9055-kube-api-access-v7j74\") pod \"26618807-5b73-42bf-ad76-b3e5c05a9055\" (UID: \"26618807-5b73-42bf-ad76-b3e5c05a9055\") "
Nov 02 08:27:03.210559 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:03.210533    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26618807-5b73-42bf-ad76-b3e5c05a9055-plugins-config-volume" pod "26618807-5b73-42bf-ad76-b3e5c05a9055" (UID: "26618807-5b73-42bf-ad76-b3e5c05a9055"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:27:03.210902 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:03.210870    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26618807-5b73-42bf-ad76-b3e5c05a9055-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:03.212397 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:03.212375    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26618807-5b73-42bf-ad76-b3e5c05a9055-kube-api-access-v7j74" pod "26618807-5b73-42bf-ad76-b3e5c05a9055" (UID: "26618807-5b73-42bf-ad76-b3e5c05a9055"). InnerVolumeSpecName "kube-api-access-v7j74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:03.312521 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:03.312451    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7j74\" (UniqueName: \"kubernetes.io/projected/26618807-5b73-42bf-ad76-b3e5c05a9055-kube-api-access-v7j74\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:04.048440 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:04.048406    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"
Nov 02 08:27:04.048440 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:04.048430    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb" event={"ID":"26618807-5b73-42bf-ad76-b3e5c05a9055","Type":"ContainerDied","Data":"4844d9cffb27539e087dcebe725df2d17c013e3ba35712508b55ae207522e41d"}
Nov 02 08:27:04.049049 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:04.048517    4898 scope.go:122] "RemoveContainer" containerID="71627ac25cd2e36f41b838aaf7d46fffc9b95293ff744c95cfb520685002673a"
Nov 02 08:27:04.102460 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:04.102432    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"]
Nov 02 08:27:04.114683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:04.114656    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d042a8-7d6f4dd8ff-nv6fb"]
Nov 02 08:27:05.470061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:05.470023    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26618807-5b73-42bf-ad76-b3e5c05a9055" path="/var/lib/kubelet/pods/26618807-5b73-42bf-ad76-b3e5c05a9055/volumes"
Nov 02 08:27:09.210985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:09.210942    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z" podUID="c640c1dc-baae-4bb0-acdb-a5b09e732e55" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:27:13.513802 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:13.513771    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"
Nov 02 08:27:13.632036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:13.631996    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c640c1dc-baae-4bb0-acdb-a5b09e732e55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c640c1dc-baae-4bb0-acdb-a5b09e732e55-plugins-config-volume\") pod \"c640c1dc-baae-4bb0-acdb-a5b09e732e55\" (UID: \"c640c1dc-baae-4bb0-acdb-a5b09e732e55\") "
Nov 02 08:27:13.632239 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:13.632115    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c640c1dc-baae-4bb0-acdb-a5b09e732e55-kube-api-access-dtpvn\" (UniqueName: \"kubernetes.io/projected/c640c1dc-baae-4bb0-acdb-a5b09e732e55-kube-api-access-dtpvn\") pod \"c640c1dc-baae-4bb0-acdb-a5b09e732e55\" (UID: \"c640c1dc-baae-4bb0-acdb-a5b09e732e55\") "
Nov 02 08:27:13.632369 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:13.632346    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c640c1dc-baae-4bb0-acdb-a5b09e732e55-plugins-config-volume" pod "c640c1dc-baae-4bb0-acdb-a5b09e732e55" (UID: "c640c1dc-baae-4bb0-acdb-a5b09e732e55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:27:13.632882 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:13.632853    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c640c1dc-baae-4bb0-acdb-a5b09e732e55-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:13.634421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:13.634396    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c640c1dc-baae-4bb0-acdb-a5b09e732e55-kube-api-access-dtpvn" pod "c640c1dc-baae-4bb0-acdb-a5b09e732e55" (UID: "c640c1dc-baae-4bb0-acdb-a5b09e732e55"). InnerVolumeSpecName "kube-api-access-dtpvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:13.734751 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:13.734671    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtpvn\" (UniqueName: \"kubernetes.io/projected/c640c1dc-baae-4bb0-acdb-a5b09e732e55-kube-api-access-dtpvn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:14.138425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:14.138332    4898 generic.go:356] "Generic (PLEG): container finished" podID="c640c1dc-baae-4bb0-acdb-a5b09e732e55" containerID="1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882" exitCode=0
Nov 02 08:27:14.138425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:14.138377    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"
Nov 02 08:27:14.138685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:14.138435    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z" event={"ID":"c640c1dc-baae-4bb0-acdb-a5b09e732e55","Type":"ContainerDied","Data":"1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882"}
Nov 02 08:27:14.138685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:14.138500    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z" event={"ID":"c640c1dc-baae-4bb0-acdb-a5b09e732e55","Type":"ContainerDied","Data":"a49a5fcc1652fe9936fd62a5610d38540a59446a89b09cbe8090dba542941011"}
Nov 02 08:27:14.138685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:14.138531    4898 scope.go:122] "RemoveContainer" containerID="1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882"
Nov 02 08:27:14.150838 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:14.150819    4898 scope.go:122] "RemoveContainer" containerID="1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882"
Nov 02 08:27:14.151118 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:27:14.151095    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882\": container with ID starting with 1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882 not found: ID does not exist" containerID="1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882"
Nov 02 08:27:14.151219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:14.151125    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882"} err="failed to get container status \"1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882\": rpc error: code = NotFound desc = could not find container \"1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882\": container with ID starting with 1dd656fd0eb2fc6d73129f4c4e5107e1d14f83c949a616fce95756e2ae048882 not found: ID does not exist"
Nov 02 08:27:14.193037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:14.193011    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"]
Nov 02 08:27:14.203922 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:14.203884    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ba44ed-65d95b88d7-5tn8z"]
Nov 02 08:27:15.036854 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:27:15.036815    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:27:15.467254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:15.467214    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c640c1dc-baae-4bb0-acdb-a5b09e732e55" path="/var/lib/kubelet/pods/c640c1dc-baae-4bb0-acdb-a5b09e732e55/volumes"
Nov 02 08:27:16.031265 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:16.031232    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"]
Nov 02 08:27:16.043123 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:16.043095    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d19e83c-th7z9"]
Nov 02 08:27:17.469851 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:17.469817    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29106bce-b602-43d7-bfbb-bd7ad1f4dc99" path="/var/lib/kubelet/pods/29106bce-b602-43d7-bfbb-bd7ad1f4dc99/volumes"
Nov 02 08:27:29.032881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:29.032844    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"]
Nov 02 08:27:29.044042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:29.044013    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f2dd91-kp46l"]
Nov 02 08:27:29.467028 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:29.466991    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63168dd-cf88-4fd3-b4a5-6e8b39089859" path="/var/lib/kubelet/pods/d63168dd-cf88-4fd3-b4a5-6e8b39089859/volumes"
Nov 02 08:27:31.708742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:31.708705    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"]
Nov 02 08:27:31.709203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:31.708991    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy" containerID="cri-o://ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec" gracePeriod=30
Nov 02 08:27:32.579883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:32.570432    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"]
Nov 02 08:27:32.579883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:32.570687    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr" podUID="7e1fd0fb-6cda-4a52-afcf-2d0698baf983" containerName="istio-proxy" containerID="cri-o://20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9" gracePeriod=30
Nov 02 08:27:37.035377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.035348    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:27:37.151759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.151718    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-podinfo\") pod \"bf9341cd-5407-4907-a11d-b3af85a6baa5\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") "
Nov 02 08:27:37.151990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.151779    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-token\" (UniqueName: \"kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-token\") pod \"bf9341cd-5407-4907-a11d-b3af85a6baa5\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") "
Nov 02 08:27:37.151990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.151858    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf9341cd-5407-4907-a11d-b3af85a6baa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9341cd-5407-4907-a11d-b3af85a6baa5-istiod-ca-cert\") pod \"bf9341cd-5407-4907-a11d-b3af85a6baa5\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") "
Nov 02 08:27:37.151990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.151948    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-socket\") pod \"bf9341cd-5407-4907-a11d-b3af85a6baa5\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") "
Nov 02 08:27:37.152191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.152009    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-data\") pod \"bf9341cd-5407-4907-a11d-b3af85a6baa5\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") "
Nov 02 08:27:37.152250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.152205    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-certs\") pod \"bf9341cd-5407-4907-a11d-b3af85a6baa5\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") "
Nov 02 08:27:37.152305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.152251    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-kube-api-access-2494k\" (UniqueName: \"kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-kube-api-access-2494k\") pod \"bf9341cd-5407-4907-a11d-b3af85a6baa5\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") "
Nov 02 08:27:37.152305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.152247    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-socket" pod "bf9341cd-5407-4907-a11d-b3af85a6baa5" (UID: "bf9341cd-5407-4907-a11d-b3af85a6baa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:37.152305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.152295    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-envoy\") pod \"bf9341cd-5407-4907-a11d-b3af85a6baa5\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") "
Nov 02 08:27:37.152463 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.152309    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-data" pod "bf9341cd-5407-4907-a11d-b3af85a6baa5" (UID: "bf9341cd-5407-4907-a11d-b3af85a6baa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:37.152463 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.152395    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9341cd-5407-4907-a11d-b3af85a6baa5-istiod-ca-cert" pod "bf9341cd-5407-4907-a11d-b3af85a6baa5" (UID: "bf9341cd-5407-4907-a11d-b3af85a6baa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:27:37.152596 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.152535    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-credential-socket\") pod \"bf9341cd-5407-4907-a11d-b3af85a6baa5\" (UID: \"bf9341cd-5407-4907-a11d-b3af85a6baa5\") "
Nov 02 08:27:37.152596 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.152547    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-certs" pod "bf9341cd-5407-4907-a11d-b3af85a6baa5" (UID: "bf9341cd-5407-4907-a11d-b3af85a6baa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:37.152789 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.152766    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-credential-socket" pod "bf9341cd-5407-4907-a11d-b3af85a6baa5" (UID: "bf9341cd-5407-4907-a11d-b3af85a6baa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:37.153342 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.153318    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9341cd-5407-4907-a11d-b3af85a6baa5-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:37.153425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.153349    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:37.153425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.153363    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:37.153425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.153377    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:37.153425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.153390    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:37.154443 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.154410    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-token" pod "bf9341cd-5407-4907-a11d-b3af85a6baa5" (UID: "bf9341cd-5407-4907-a11d-b3af85a6baa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:37.154828 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.154801    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-kube-api-access-2494k" pod "bf9341cd-5407-4907-a11d-b3af85a6baa5" (UID: "bf9341cd-5407-4907-a11d-b3af85a6baa5"). InnerVolumeSpecName "kube-api-access-2494k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:37.154933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.154824    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-podinfo" pod "bf9341cd-5407-4907-a11d-b3af85a6baa5" (UID: "bf9341cd-5407-4907-a11d-b3af85a6baa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:27:37.155006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.154966    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-envoy" pod "bf9341cd-5407-4907-a11d-b3af85a6baa5" (UID: "bf9341cd-5407-4907-a11d-b3af85a6baa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:37.254954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.254842    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2494k\" (UniqueName: \"kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-kube-api-access-2494k\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:37.254954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.254873    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:37.254954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.254882    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:37.254954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.254937    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9341cd-5407-4907-a11d-b3af85a6baa5-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:37.517608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.517527    4898 generic.go:356] "Generic (PLEG): container finished" podID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerID="ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec" exitCode=0
Nov 02 08:27:37.517608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.517579    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"
Nov 02 08:27:37.517798 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.517598    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" event={"ID":"bf9341cd-5407-4907-a11d-b3af85a6baa5","Type":"ContainerDied","Data":"ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec"}
Nov 02 08:27:37.517798 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.517661    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm" event={"ID":"bf9341cd-5407-4907-a11d-b3af85a6baa5","Type":"ContainerDied","Data":"ee0a2f7b7f2b030fc5adc14dbba5be9578cf71156421f96cb1f63800a7fd0f8f"}
Nov 02 08:27:37.517798 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.517693    4898 scope.go:122] "RemoveContainer" containerID="ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec"
Nov 02 08:27:37.530383 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.530360    4898 scope.go:122] "RemoveContainer" containerID="681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830"
Nov 02 08:27:37.540799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.540778    4898 scope.go:122] "RemoveContainer" containerID="ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec"
Nov 02 08:27:37.541098 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:27:37.541077    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec\": container with ID starting with ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec not found: ID does not exist" containerID="ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec"
Nov 02 08:27:37.541177 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.541106    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec"} err="failed to get container status \"ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec\": rpc error: code = NotFound desc = could not find container \"ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec\": container with ID starting with ab303cf46216ac7ae605ae8970568b9309879d2825ba0327a60f258c314dd5ec not found: ID does not exist"
Nov 02 08:27:37.541177 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.541122    4898 scope.go:122] "RemoveContainer" containerID="681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830"
Nov 02 08:27:37.541376 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:27:37.541355    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830\": container with ID starting with 681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830 not found: ID does not exist" containerID="681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830"
Nov 02 08:27:37.541427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.541381    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830"} err="failed to get container status \"681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830\": rpc error: code = NotFound desc = could not find container \"681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830\": container with ID starting with 681c667f9f6eebe22e782e4fbe86ff68b4a2c1e5cd98e5e256377cbd3b29e830 not found: ID does not exist"
Nov 02 08:27:37.572740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.572715    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"]
Nov 02 08:27:37.585054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.585030    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b72fa3-openshift-default-7867f84765-4ldbm"]
Nov 02 08:27:37.896820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:37.896787    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:27:38.075845 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.075816    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-token\" (UniqueName: \"kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-token\") pod \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") "
Nov 02 08:27:38.076346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.075867    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-socket\") pod \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") "
Nov 02 08:27:38.076346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.076146    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-podinfo\") pod \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") "
Nov 02 08:27:38.076346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.076202    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-kube-api-access-d4frj\" (UniqueName: \"kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-kube-api-access-d4frj\") pod \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") "
Nov 02 08:27:38.076346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.076238    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-certs\") pod \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") "
Nov 02 08:27:38.076346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.076273    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istiod-ca-cert\") pod \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") "
Nov 02 08:27:38.076346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.076271    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-socket" pod "7e1fd0fb-6cda-4a52-afcf-2d0698baf983" (UID: "7e1fd0fb-6cda-4a52-afcf-2d0698baf983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:38.076346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.076300    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-envoy\") pod \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") "
Nov 02 08:27:38.076785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.076387    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-data\") pod \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") "
Nov 02 08:27:38.076785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.076461    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-credential-socket\") pod \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\" (UID: \"7e1fd0fb-6cda-4a52-afcf-2d0698baf983\") "
Nov 02 08:27:38.076785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.076531    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-certs" pod "7e1fd0fb-6cda-4a52-afcf-2d0698baf983" (UID: "7e1fd0fb-6cda-4a52-afcf-2d0698baf983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:38.076785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.076754    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-data" pod "7e1fd0fb-6cda-4a52-afcf-2d0698baf983" (UID: "7e1fd0fb-6cda-4a52-afcf-2d0698baf983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:38.077041 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.076928    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-credential-socket" pod "7e1fd0fb-6cda-4a52-afcf-2d0698baf983" (UID: "7e1fd0fb-6cda-4a52-afcf-2d0698baf983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:38.077041 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.076950    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istiod-ca-cert" pod "7e1fd0fb-6cda-4a52-afcf-2d0698baf983" (UID: "7e1fd0fb-6cda-4a52-afcf-2d0698baf983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:27:38.077275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.077252    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.077353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.077280    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.077353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.077293    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.077353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.077305    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.077353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.077320    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.078684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.078661    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-podinfo" pod "7e1fd0fb-6cda-4a52-afcf-2d0698baf983" (UID: "7e1fd0fb-6cda-4a52-afcf-2d0698baf983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:27:38.078787 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.078741    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-token" pod "7e1fd0fb-6cda-4a52-afcf-2d0698baf983" (UID: "7e1fd0fb-6cda-4a52-afcf-2d0698baf983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:38.078883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.078867    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-kube-api-access-d4frj" pod "7e1fd0fb-6cda-4a52-afcf-2d0698baf983" (UID: "7e1fd0fb-6cda-4a52-afcf-2d0698baf983"). InnerVolumeSpecName "kube-api-access-d4frj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:38.079040 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.079023    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-envoy" pod "7e1fd0fb-6cda-4a52-afcf-2d0698baf983" (UID: "7e1fd0fb-6cda-4a52-afcf-2d0698baf983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:38.194170 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.183191    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.194170 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.183226    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.194170 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.183241    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4frj\" (UniqueName: \"kubernetes.io/projected/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-kube-api-access-d4frj\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.194170 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.183259    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1fd0fb-6cda-4a52-afcf-2d0698baf983-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:38.576913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.572413    4898 generic.go:356] "Generic (PLEG): container finished" podID="7e1fd0fb-6cda-4a52-afcf-2d0698baf983" containerID="20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9" exitCode=0
Nov 02 08:27:38.576913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.572559    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr" event={"ID":"7e1fd0fb-6cda-4a52-afcf-2d0698baf983","Type":"ContainerDied","Data":"20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9"}
Nov 02 08:27:38.576913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.572614    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr" event={"ID":"7e1fd0fb-6cda-4a52-afcf-2d0698baf983","Type":"ContainerDied","Data":"5c040e1c8615feb6cea672816d63e364fc9546d1c2a1e998ee86f113e2a38831"}
Nov 02 08:27:38.576913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.572642    4898 scope.go:122] "RemoveContainer" containerID="20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9"
Nov 02 08:27:38.576913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.572748    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"
Nov 02 08:27:38.651103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.651005    4898 scope.go:122] "RemoveContainer" containerID="20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9"
Nov 02 08:27:38.660940 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:27:38.651497    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9\": container with ID starting with 20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9 not found: ID does not exist" containerID="20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9"
Nov 02 08:27:38.660940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.651548    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9"} err="failed to get container status \"20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9\": rpc error: code = NotFound desc = could not find container \"20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9\": container with ID starting with 20a7fd8d7f1ace7286421ab7ce4ed690f82d7f26c2666bb4ed38ba11f2f2f5c9 not found: ID does not exist"
Nov 02 08:27:38.721825 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.721788    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"]
Nov 02 08:27:38.726426 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:38.726399    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d042a8-openshift-default-56bb565455-kngvr"]
Nov 02 08:27:39.474586 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:39.474551    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1fd0fb-6cda-4a52-afcf-2d0698baf983" path="/var/lib/kubelet/pods/7e1fd0fb-6cda-4a52-afcf-2d0698baf983/volumes"
Nov 02 08:27:39.475272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:39.475249    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" path="/var/lib/kubelet/pods/bf9341cd-5407-4907-a11d-b3af85a6baa5/volumes"
Nov 02 08:27:41.705055 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:41.705022    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"]
Nov 02 08:27:41.705567 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:41.705290    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy" containerID="cri-o://61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64" gracePeriod=30
Nov 02 08:27:43.340270 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:43.340229    4898 patch_prober.go:29] interesting pod/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:27:43.340664 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:43.340320    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:27:44.994803 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:27:44.994761    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:27:45.797978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.797944    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"]
Nov 02 08:27:45.799029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799004    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43" containerName="revoke-keys"
Nov 02 08:27:45.799029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799029    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43" containerName="revoke-keys"
Nov 02 08:27:45.799211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799048    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy"
Nov 02 08:27:45.799211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799054    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy"
Nov 02 08:27:45.799211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799068    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8" containerName="payload-processing"
Nov 02 08:27:45.799211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799076    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8" containerName="payload-processing"
Nov 02 08:27:45.799211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799110    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy"
Nov 02 08:27:45.799211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799117    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy"
Nov 02 08:27:45.799211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799150    4898 cpu_manager.go:414] "containerMap: removing container" podUID="26618807-5b73-42bf-ad76-b3e5c05a9055" containerName="payload-pre-processing"
Nov 02 08:27:45.799211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799156    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26618807-5b73-42bf-ad76-b3e5c05a9055" containerName="payload-pre-processing"
Nov 02 08:27:45.799211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799169    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c640c1dc-baae-4bb0-acdb-a5b09e732e55" containerName="payload-pre-processing"
Nov 02 08:27:45.799211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799174    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c640c1dc-baae-4bb0-acdb-a5b09e732e55" containerName="payload-pre-processing"
Nov 02 08:27:45.799211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799192    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7e1fd0fb-6cda-4a52-afcf-2d0698baf983" containerName="istio-proxy"
Nov 02 08:27:45.799211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799196    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1fd0fb-6cda-4a52-afcf-2d0698baf983" containerName="istio-proxy"
Nov 02 08:27:45.799867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799316    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1fd0fb-6cda-4a52-afcf-2d0698baf983" containerName="istio-proxy"
Nov 02 08:27:45.799867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799326    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43" containerName="revoke-keys"
Nov 02 08:27:45.799867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799332    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy"
Nov 02 08:27:45.799867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799338    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2428c6e0-6a9e-4bf3-bc9d-9ec82d8afab8" containerName="payload-processing"
Nov 02 08:27:45.799867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799348    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy"
Nov 02 08:27:45.799867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799357    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="26618807-5b73-42bf-ad76-b3e5c05a9055" containerName="payload-pre-processing"
Nov 02 08:27:45.799867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799364    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy"
Nov 02 08:27:45.799867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.799370    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c640c1dc-baae-4bb0-acdb-a5b09e732e55" containerName="payload-pre-processing"
Nov 02 08:27:45.804510 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.804489    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:45.819765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.819735    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"]
Nov 02 08:27:45.827258 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.827228    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:27:45.827396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.827333    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:27:45.827396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.827334    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:27:45.831566 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.831529    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:27:45.947037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.947000    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e21c098c-45s6h\" (UID: \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:45.947221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.947056    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6hl8\" (UniqueName: \"kubernetes.io/projected/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-kube-api-access-f6hl8\") pod \"maas-api-revoke-sub-e21c098c-45s6h\" (UID: \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:45.947221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:45.947191    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e21c098c-45s6h\" (UID: \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:46.057506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:46.054937    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e21c098c-45s6h\" (UID: \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:46.057506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:46.055014    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hl8\" (UniqueName: \"kubernetes.io/projected/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-kube-api-access-f6hl8\") pod \"maas-api-revoke-sub-e21c098c-45s6h\" (UID: \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:46.057506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:46.055111    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e21c098c-45s6h\" (UID: \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:46.062084 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:46.059724    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e21c098c-45s6h\" (UID: \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:46.095914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:46.095187    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e21c098c-45s6h\" (UID: \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:46.099912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:46.099456    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hl8\" (UniqueName: \"kubernetes.io/projected/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-kube-api-access-f6hl8\") pod \"maas-api-revoke-sub-e21c098c-45s6h\" (UID: \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:46.124993 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:46.123145    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:46.457392 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:27:46.457351    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048db14a_1ff5_4e5b_84f1_6b8ab2160e7e.slice/crio-2a47e23ddbbb79ba2d2790cbe52ea4469d1313bfb81480d2e2819919b0393709 WatchSource:0}: Error finding container 2a47e23ddbbb79ba2d2790cbe52ea4469d1313bfb81480d2e2819919b0393709: Status 404 returned error can't find the container with id 2a47e23ddbbb79ba2d2790cbe52ea4469d1313bfb81480d2e2819919b0393709
Nov 02 08:27:46.459541 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:46.459509    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"]
Nov 02 08:27:46.738410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:46.738378    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h" event={"ID":"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e","Type":"ContainerStarted","Data":"2a47e23ddbbb79ba2d2790cbe52ea4469d1313bfb81480d2e2819919b0393709"}
Nov 02 08:27:46.978868 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:46.978794    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:27:47.072250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.072220    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-token\" (UniqueName: \"kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-token\") pod \"dcce1599-5df4-4e0a-853a-6127ac605ffe\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") "
Nov 02 08:27:47.072740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.072346    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-socket\") pod \"dcce1599-5df4-4e0a-853a-6127ac605ffe\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") "
Nov 02 08:27:47.072740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.072459    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-podinfo\") pod \"dcce1599-5df4-4e0a-853a-6127ac605ffe\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") "
Nov 02 08:27:47.072740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.072534    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-data\") pod \"dcce1599-5df4-4e0a-853a-6127ac605ffe\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") "
Nov 02 08:27:47.072740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.072594    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-credential-socket\") pod \"dcce1599-5df4-4e0a-853a-6127ac605ffe\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") "
Nov 02 08:27:47.072740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.072622    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-certs\") pod \"dcce1599-5df4-4e0a-853a-6127ac605ffe\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") "
Nov 02 08:27:47.072740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.072666    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-envoy\") pod \"dcce1599-5df4-4e0a-853a-6127ac605ffe\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") "
Nov 02 08:27:47.072740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.072678    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-socket" pod "dcce1599-5df4-4e0a-853a-6127ac605ffe" (UID: "dcce1599-5df4-4e0a-853a-6127ac605ffe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:47.073130 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.072773    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcce1599-5df4-4e0a-853a-6127ac605ffe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcce1599-5df4-4e0a-853a-6127ac605ffe-istiod-ca-cert\") pod \"dcce1599-5df4-4e0a-853a-6127ac605ffe\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") "
Nov 02 08:27:47.073130 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.072818    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-kube-api-access-gkfnq\" (UniqueName: \"kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-kube-api-access-gkfnq\") pod \"dcce1599-5df4-4e0a-853a-6127ac605ffe\" (UID: \"dcce1599-5df4-4e0a-853a-6127ac605ffe\") "
Nov 02 08:27:47.073130 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.072972    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-credential-socket" pod "dcce1599-5df4-4e0a-853a-6127ac605ffe" (UID: "dcce1599-5df4-4e0a-853a-6127ac605ffe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:47.073299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.073148    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-data" pod "dcce1599-5df4-4e0a-853a-6127ac605ffe" (UID: "dcce1599-5df4-4e0a-853a-6127ac605ffe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:47.073652 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.073570    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.073652 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.073597    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.073652 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.073611    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.073652 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.073620    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-certs" pod "dcce1599-5df4-4e0a-853a-6127ac605ffe" (UID: "dcce1599-5df4-4e0a-853a-6127ac605ffe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:47.074290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.073988    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcce1599-5df4-4e0a-853a-6127ac605ffe-istiod-ca-cert" pod "dcce1599-5df4-4e0a-853a-6127ac605ffe" (UID: "dcce1599-5df4-4e0a-853a-6127ac605ffe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:27:47.076434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.076404    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-kube-api-access-gkfnq" pod "dcce1599-5df4-4e0a-853a-6127ac605ffe" (UID: "dcce1599-5df4-4e0a-853a-6127ac605ffe"). InnerVolumeSpecName "kube-api-access-gkfnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:47.076754 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.076729    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-token" pod "dcce1599-5df4-4e0a-853a-6127ac605ffe" (UID: "dcce1599-5df4-4e0a-853a-6127ac605ffe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:47.076853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.076814    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-podinfo" pod "dcce1599-5df4-4e0a-853a-6127ac605ffe" (UID: "dcce1599-5df4-4e0a-853a-6127ac605ffe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:27:47.078554 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.078528    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-envoy" pod "dcce1599-5df4-4e0a-853a-6127ac605ffe" (UID: "dcce1599-5df4-4e0a-853a-6127ac605ffe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:27:47.174985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.174955    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.174985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.174984    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.174985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.174993    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcce1599-5df4-4e0a-853a-6127ac605ffe-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.175201 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.175002    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkfnq\" (UniqueName: \"kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-kube-api-access-gkfnq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.175201 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.175013    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.175201 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.175021    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcce1599-5df4-4e0a-853a-6127ac605ffe-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:47.748402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.748364    4898 generic.go:356] "Generic (PLEG): container finished" podID="048db14a-1ff5-4e5b-84f1-6b8ab2160e7e" containerID="5c052a387ad77177419701fd6f42f4d1ef63921e25a83366b52d0519875ae068" exitCode=0
Nov 02 08:27:47.748604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.748455    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h" event={"ID":"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e","Type":"ContainerDied","Data":"5c052a387ad77177419701fd6f42f4d1ef63921e25a83366b52d0519875ae068"}
Nov 02 08:27:47.750459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.750433    4898 generic.go:356] "Generic (PLEG): container finished" podID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerID="61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64" exitCode=0
Nov 02 08:27:47.750574 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.750485    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"
Nov 02 08:27:47.750574 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.750520    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" event={"ID":"dcce1599-5df4-4e0a-853a-6127ac605ffe","Type":"ContainerDied","Data":"61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64"}
Nov 02 08:27:47.750574 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.750570    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl" event={"ID":"dcce1599-5df4-4e0a-853a-6127ac605ffe","Type":"ContainerDied","Data":"2605480c11c97fdd3b899ff091c422592935a951e666a6753fe7a621f486c750"}
Nov 02 08:27:47.750684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.750597    4898 scope.go:122] "RemoveContainer" containerID="61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64"
Nov 02 08:27:47.763031 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.763011    4898 scope.go:122] "RemoveContainer" containerID="4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b"
Nov 02 08:27:47.773693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.773669    4898 scope.go:122] "RemoveContainer" containerID="61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64"
Nov 02 08:27:47.774023 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:27:47.773999    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64\": container with ID starting with 61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64 not found: ID does not exist" containerID="61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64"
Nov 02 08:27:47.774092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.774035    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64"} err="failed to get container status \"61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64\": rpc error: code = NotFound desc = could not find container \"61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64\": container with ID starting with 61b5a5518242ece2394c54f209d650bd576919c47aed9d9d0893cc7388f2ca64 not found: ID does not exist"
Nov 02 08:27:47.774092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.774057    4898 scope.go:122] "RemoveContainer" containerID="4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b"
Nov 02 08:27:47.774312 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:27:47.774293    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b\": container with ID starting with 4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b not found: ID does not exist" containerID="4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b"
Nov 02 08:27:47.774364 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.774317    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b"} err="failed to get container status \"4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b\": rpc error: code = NotFound desc = could not find container \"4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b\": container with ID starting with 4dcbab583b660661b0dd346daeccef4e5b6f7c12a28902b959d86f56468e6e0b not found: ID does not exist"
Nov 02 08:27:47.870009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.869962    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"]
Nov 02 08:27:47.876180 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:47.876133    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ba44ed-openshift-default-69b4f8b76f-2wltl"]
Nov 02 08:27:48.928824 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:48.928797    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:49.002659 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.002617    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-xks-ca\") pod \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\" (UID: \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\") "
Nov 02 08:27:49.002854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.002726    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-openshift-ca\") pod \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\" (UID: \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\") "
Nov 02 08:27:49.002854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.002804    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-kube-api-access-f6hl8\" (UniqueName: \"kubernetes.io/projected/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-kube-api-access-f6hl8\") pod \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\" (UID: \"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e\") "
Nov 02 08:27:49.004110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.004080    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-openshift-ca" pod "048db14a-1ff5-4e5b-84f1-6b8ab2160e7e" (UID: "048db14a-1ff5-4e5b-84f1-6b8ab2160e7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:27:49.005798 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.005766    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-kube-api-access-f6hl8" pod "048db14a-1ff5-4e5b-84f1-6b8ab2160e7e" (UID: "048db14a-1ff5-4e5b-84f1-6b8ab2160e7e"). InnerVolumeSpecName "kube-api-access-f6hl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:27:49.006249 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.006226    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-xks-ca" pod "048db14a-1ff5-4e5b-84f1-6b8ab2160e7e" (UID: "048db14a-1ff5-4e5b-84f1-6b8ab2160e7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:27:49.104570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.104529    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:49.104570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.104565    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:49.104745 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.104580    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6hl8\" (UniqueName: \"kubernetes.io/projected/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e-kube-api-access-f6hl8\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:27:49.474815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.474774    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" path="/var/lib/kubelet/pods/dcce1599-5df4-4e0a-853a-6127ac605ffe/volumes"
Nov 02 08:27:49.770718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.770619    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"
Nov 02 08:27:49.770718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.770686    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h" event={"ID":"048db14a-1ff5-4e5b-84f1-6b8ab2160e7e","Type":"ContainerDied","Data":"2a47e23ddbbb79ba2d2790cbe52ea4469d1313bfb81480d2e2819919b0393709"}
Nov 02 08:27:49.771020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:27:49.770748    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a47e23ddbbb79ba2d2790cbe52ea4469d1313bfb81480d2e2819919b0393709"
Nov 02 08:28:14.882586 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:14.882507    4898 scope.go:122] "RemoveContainer" containerID="5909188927c5761741d9b3dd384bb173fc593fd17be979d4af0ea17dc31c3eb4"
Nov 02 08:28:14.893738 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:14.893716    4898 scope.go:122] "RemoveContainer" containerID="893838fd42863099c00dbf7e7fef18842e24579c95c53ea9b37e124e2be0be8e"
Nov 02 08:28:14.903970 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:14.903947    4898 scope.go:122] "RemoveContainer" containerID="3409446a3c879302a2390d8fb8893b0bfbd9278fbf1b0ff34b0f7d9822dee506"
Nov 02 08:28:14.989414 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:28:14.989380    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:28:23.597340 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:23.597290    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"]
Nov 02 08:28:23.598493 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:23.598458    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy" containerID="cri-o://b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f" gracePeriod=30
Nov 02 08:28:27.694214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:27.694170    4898 patch_prober.go:29] interesting pod/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:28:27.694614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:27.694246    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:28:28.892758 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.892696    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:28:28.964676 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.964632    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-envoy\") pod \"9474d0bd-4733-44e8-a308-e15f2edb4108\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") "
Nov 02 08:28:28.964855 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.964700    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-socket\") pod \"9474d0bd-4733-44e8-a308-e15f2edb4108\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") "
Nov 02 08:28:28.964855 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.964843    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-data\") pod \"9474d0bd-4733-44e8-a308-e15f2edb4108\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") "
Nov 02 08:28:28.964996 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.964961    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-credential-socket\") pod \"9474d0bd-4733-44e8-a308-e15f2edb4108\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") "
Nov 02 08:28:28.965061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.964995    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-podinfo\") pod \"9474d0bd-4733-44e8-a308-e15f2edb4108\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") "
Nov 02 08:28:28.965105 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.965046    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-socket" pod "9474d0bd-4733-44e8-a308-e15f2edb4108" (UID: "9474d0bd-4733-44e8-a308-e15f2edb4108"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:28:28.965105 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.965071    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-token\" (UniqueName: \"kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-token\") pod \"9474d0bd-4733-44e8-a308-e15f2edb4108\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") "
Nov 02 08:28:28.965182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.965154    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-certs\") pod \"9474d0bd-4733-44e8-a308-e15f2edb4108\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") "
Nov 02 08:28:28.965243 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.965178    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9474d0bd-4733-44e8-a308-e15f2edb4108-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9474d0bd-4733-44e8-a308-e15f2edb4108-istiod-ca-cert\") pod \"9474d0bd-4733-44e8-a308-e15f2edb4108\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") "
Nov 02 08:28:28.965305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.965266    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-kube-api-access-pwqfp\" (UniqueName: \"kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-kube-api-access-pwqfp\") pod \"9474d0bd-4733-44e8-a308-e15f2edb4108\" (UID: \"9474d0bd-4733-44e8-a308-e15f2edb4108\") "
Nov 02 08:28:28.965305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.965281    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-credential-socket" pod "9474d0bd-4733-44e8-a308-e15f2edb4108" (UID: "9474d0bd-4733-44e8-a308-e15f2edb4108"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:28:28.965505 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.965485    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-data" pod "9474d0bd-4733-44e8-a308-e15f2edb4108" (UID: "9474d0bd-4733-44e8-a308-e15f2edb4108"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:28:28.965882 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.965857    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:28:28.966030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.965881    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:28:28.966030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.965931    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:28:28.966253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.966232    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-certs" pod "9474d0bd-4733-44e8-a308-e15f2edb4108" (UID: "9474d0bd-4733-44e8-a308-e15f2edb4108"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:28:28.966519 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.966483    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9474d0bd-4733-44e8-a308-e15f2edb4108-istiod-ca-cert" pod "9474d0bd-4733-44e8-a308-e15f2edb4108" (UID: "9474d0bd-4733-44e8-a308-e15f2edb4108"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:28:28.967673 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.967648    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-envoy" pod "9474d0bd-4733-44e8-a308-e15f2edb4108" (UID: "9474d0bd-4733-44e8-a308-e15f2edb4108"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:28:28.968420 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.968394    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-token" pod "9474d0bd-4733-44e8-a308-e15f2edb4108" (UID: "9474d0bd-4733-44e8-a308-e15f2edb4108"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:28:28.968542 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.968456    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-podinfo" pod "9474d0bd-4733-44e8-a308-e15f2edb4108" (UID: "9474d0bd-4733-44e8-a308-e15f2edb4108"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:28:28.972986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:28.972961    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-kube-api-access-pwqfp" pod "9474d0bd-4733-44e8-a308-e15f2edb4108" (UID: "9474d0bd-4733-44e8-a308-e15f2edb4108"). InnerVolumeSpecName "kube-api-access-pwqfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:28:29.067000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.066970    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:28:29.067000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.066997    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:28:29.067198 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.067012    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:28:29.067198 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.067021    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9474d0bd-4733-44e8-a308-e15f2edb4108-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:28:29.067198 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.067029    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwqfp\" (UniqueName: \"kubernetes.io/projected/9474d0bd-4733-44e8-a308-e15f2edb4108-kube-api-access-pwqfp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:28:29.067198 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.067040    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9474d0bd-4733-44e8-a308-e15f2edb4108-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:28:29.248248 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.248215    4898 generic.go:356] "Generic (PLEG): container finished" podID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerID="b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f" exitCode=0
Nov 02 08:28:29.248437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.248279    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"
Nov 02 08:28:29.248437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.248303    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" event={"ID":"9474d0bd-4733-44e8-a308-e15f2edb4108","Type":"ContainerDied","Data":"b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f"}
Nov 02 08:28:29.248437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.248357    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs" event={"ID":"9474d0bd-4733-44e8-a308-e15f2edb4108","Type":"ContainerDied","Data":"c57106053f7e378c93d8088a3c7b1d15004d1ca8e4ef1b420b156fea3c240fb8"}
Nov 02 08:28:29.248437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.248381    4898 scope.go:122] "RemoveContainer" containerID="b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f"
Nov 02 08:28:29.261988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.261875    4898 scope.go:122] "RemoveContainer" containerID="58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a"
Nov 02 08:28:29.282753 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.282727    4898 scope.go:122] "RemoveContainer" containerID="b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f"
Nov 02 08:28:29.283132 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:28:29.283109    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f\": container with ID starting with b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f not found: ID does not exist" containerID="b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f"
Nov 02 08:28:29.283236 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.283139    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f"} err="failed to get container status \"b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f\": rpc error: code = NotFound desc = could not find container \"b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f\": container with ID starting with b18fdbcf9685a8db3aec3a1ce4388694ee2c5e4463ce7f1576dff3d32b81552f not found: ID does not exist"
Nov 02 08:28:29.283292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.283238    4898 scope.go:122] "RemoveContainer" containerID="58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a"
Nov 02 08:28:29.283545 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:28:29.283519    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a\": container with ID starting with 58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a not found: ID does not exist" containerID="58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a"
Nov 02 08:28:29.283639 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.283556    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a"} err="failed to get container status \"58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a\": rpc error: code = NotFound desc = could not find container \"58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a\": container with ID starting with 58febb4fecde6451e9189b0c3f1214e1fbaf45a729a6962ce52e01e600e4a62a not found: ID does not exist"
Nov 02 08:28:29.307361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.307337    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"]
Nov 02 08:28:29.311325 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.311300    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-eb6010-openshift-default-787f9bd597-hgrvs"]
Nov 02 08:28:29.467851 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:29.467819    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" path="/var/lib/kubelet/pods/9474d0bd-4733-44e8-a308-e15f2edb4108/volumes"
Nov 02 08:28:37.100983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:37.100948    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"]
Nov 02 08:28:42.101555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:42.101503    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r" podUID="c704b690-8c9a-4a5f-8651-848a5bbba59a" containerName="payload-pre-processing" containerID="cri-o://bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449" gracePeriod=25
Nov 02 08:28:44.991225 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:28:44.991179    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:28:46.634026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:46.633984    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r" podUID="c704b690-8c9a-4a5f-8651-848a5bbba59a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:28:56.633819 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:28:56.633781    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r" podUID="c704b690-8c9a-4a5f-8651-848a5bbba59a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:29:02.416948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.416912    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"
Nov 02 08:29:02.578551 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.578514    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c704b690-8c9a-4a5f-8651-848a5bbba59a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c704b690-8c9a-4a5f-8651-848a5bbba59a-plugins-config-volume\") pod \"c704b690-8c9a-4a5f-8651-848a5bbba59a\" (UID: \"c704b690-8c9a-4a5f-8651-848a5bbba59a\") "
Nov 02 08:29:02.578761 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.578570    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c704b690-8c9a-4a5f-8651-848a5bbba59a-kube-api-access-w58q2\" (UniqueName: \"kubernetes.io/projected/c704b690-8c9a-4a5f-8651-848a5bbba59a-kube-api-access-w58q2\") pod \"c704b690-8c9a-4a5f-8651-848a5bbba59a\" (UID: \"c704b690-8c9a-4a5f-8651-848a5bbba59a\") "
Nov 02 08:29:02.579002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.578974    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c704b690-8c9a-4a5f-8651-848a5bbba59a-plugins-config-volume" pod "c704b690-8c9a-4a5f-8651-848a5bbba59a" (UID: "c704b690-8c9a-4a5f-8651-848a5bbba59a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:29:02.579174 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.579154    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c704b690-8c9a-4a5f-8651-848a5bbba59a-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:29:02.580827 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.580806    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c704b690-8c9a-4a5f-8651-848a5bbba59a-kube-api-access-w58q2" pod "c704b690-8c9a-4a5f-8651-848a5bbba59a" (UID: "c704b690-8c9a-4a5f-8651-848a5bbba59a"). InnerVolumeSpecName "kube-api-access-w58q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:29:02.681179 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.680900    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w58q2\" (UniqueName: \"kubernetes.io/projected/c704b690-8c9a-4a5f-8651-848a5bbba59a-kube-api-access-w58q2\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:29:02.682333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.682208    4898 generic.go:356] "Generic (PLEG): container finished" podID="c704b690-8c9a-4a5f-8651-848a5bbba59a" containerID="bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449" exitCode=0
Nov 02 08:29:02.682882 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.682791    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r" event={"ID":"c704b690-8c9a-4a5f-8651-848a5bbba59a","Type":"ContainerDied","Data":"bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449"}
Nov 02 08:29:02.683107 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.683040    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r" event={"ID":"c704b690-8c9a-4a5f-8651-848a5bbba59a","Type":"ContainerDied","Data":"e0650cab8991840918ee1eaf255d8d219c8d1c0ffb8370721110725ef0ae809c"}
Nov 02 08:29:02.683290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.683087    4898 scope.go:122] "RemoveContainer" containerID="bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449"
Nov 02 08:29:02.684511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.683544    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"
Nov 02 08:29:02.702989 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.702883    4898 scope.go:122] "RemoveContainer" containerID="bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449"
Nov 02 08:29:02.703302 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:29:02.703282    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449\": container with ID starting with bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449 not found: ID does not exist" containerID="bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449"
Nov 02 08:29:02.703386 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.703312    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449"} err="failed to get container status \"bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449\": rpc error: code = NotFound desc = could not find container \"bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449\": container with ID starting with bec0acba9143c5d3257f3f859d9e5e4a3fd559f25ccebf4be73618a6ce229449 not found: ID does not exist"
Nov 02 08:29:02.740983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.740947    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"]
Nov 02 08:29:02.748207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:02.748158    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-736f7c-b6945b94c-wxz8r"]
Nov 02 08:29:03.466312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:03.466277    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c704b690-8c9a-4a5f-8651-848a5bbba59a" path="/var/lib/kubelet/pods/c704b690-8c9a-4a5f-8651-848a5bbba59a/volumes"
Nov 02 08:29:10.788124 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:10.788080    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:29:10.794036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:10.794012    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:29:14.966082 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:14.966046    4898 scope.go:122] "RemoveContainer" containerID="6e50e10225607f1ee60bbfa2fffbb617915a328d10d5cef4b0abac2880568279"
Nov 02 08:29:14.991571 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:29:14.991534    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:29:26.442996 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.442751    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"]
Nov 02 08:29:26.477736 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.477693    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"]
Nov 02 08:29:26.968248 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.968210    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"]
Nov 02 08:29:26.968922 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.968880    4898 cpu_manager.go:414] "containerMap: removing container" podUID="048db14a-1ff5-4e5b-84f1-6b8ab2160e7e" containerName="revoke-keys"
Nov 02 08:29:26.968922 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.968911    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048db14a-1ff5-4e5b-84f1-6b8ab2160e7e" containerName="revoke-keys"
Nov 02 08:29:26.968922 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.968922    4898 cpu_manager.go:414] "containerMap: removing container" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy"
Nov 02 08:29:26.969094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.968927    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy"
Nov 02 08:29:26.969094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.968944    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy"
Nov 02 08:29:26.969094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.968950    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy"
Nov 02 08:29:26.969094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.968964    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy"
Nov 02 08:29:26.969094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.968969    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy"
Nov 02 08:29:26.969094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.968995    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy"
Nov 02 08:29:26.969094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.969000    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed94547d-f04f-4401-a356-fd38ffdcf2e4" containerName="istio-proxy"
Nov 02 08:29:26.969094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.969022    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c704b690-8c9a-4a5f-8651-848a5bbba59a" containerName="payload-pre-processing"
Nov 02 08:29:26.969094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.969028    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c704b690-8c9a-4a5f-8651-848a5bbba59a" containerName="payload-pre-processing"
Nov 02 08:29:26.969402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.969151    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c704b690-8c9a-4a5f-8651-848a5bbba59a" containerName="payload-pre-processing"
Nov 02 08:29:26.969402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.969167    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="048db14a-1ff5-4e5b-84f1-6b8ab2160e7e" containerName="revoke-keys"
Nov 02 08:29:26.969402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.969172    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy"
Nov 02 08:29:26.969402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.969180    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy"
Nov 02 08:29:26.969402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.969189    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9341cd-5407-4907-a11d-b3af85a6baa5" containerName="istio-proxy"
Nov 02 08:29:26.969402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.969197    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy"
Nov 02 08:29:26.969402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.969204    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy"
Nov 02 08:29:26.973008 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.972986    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:26.988199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.988167    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:29:26.991008 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.990983    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"]
Nov 02 08:29:26.992634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.992610    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:29:26.992634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:26.992610    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:29:27.099913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:27.092230    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:29:27.148924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:27.140796    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2127178e-36c3-47c4-b407-c74773470512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bada141-pbswd\" (UID: \"2127178e-36c3-47c4-b407-c74773470512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:27.163917 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:27.152402    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2127178e-36c3-47c4-b407-c74773470512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bada141-pbswd\" (UID: \"2127178e-36c3-47c4-b407-c74773470512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:27.164270 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:27.164235    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4p6s\" (UniqueName: \"kubernetes.io/projected/2127178e-36c3-47c4-b407-c74773470512-kube-api-access-d4p6s\") pod \"maas-api-revoke-sub-1bada141-pbswd\" (UID: \"2127178e-36c3-47c4-b407-c74773470512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:27.266285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:27.266181    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4p6s\" (UniqueName: \"kubernetes.io/projected/2127178e-36c3-47c4-b407-c74773470512-kube-api-access-d4p6s\") pod \"maas-api-revoke-sub-1bada141-pbswd\" (UID: \"2127178e-36c3-47c4-b407-c74773470512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:27.266285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:27.266266    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2127178e-36c3-47c4-b407-c74773470512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bada141-pbswd\" (UID: \"2127178e-36c3-47c4-b407-c74773470512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:27.266579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:27.266554    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2127178e-36c3-47c4-b407-c74773470512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bada141-pbswd\" (UID: \"2127178e-36c3-47c4-b407-c74773470512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:27.267044 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:27.267024    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2127178e-36c3-47c4-b407-c74773470512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bada141-pbswd\" (UID: \"2127178e-36c3-47c4-b407-c74773470512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:27.269061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:27.269037    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2127178e-36c3-47c4-b407-c74773470512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bada141-pbswd\" (UID: \"2127178e-36c3-47c4-b407-c74773470512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:27.290073 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:27.290045    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4p6s\" (UniqueName: \"kubernetes.io/projected/2127178e-36c3-47c4-b407-c74773470512-kube-api-access-d4p6s\") pod \"maas-api-revoke-sub-1bada141-pbswd\" (UID: \"2127178e-36c3-47c4-b407-c74773470512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:27.588800 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:27.588710    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:27.963781 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:29:27.963733    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2127178e_36c3_47c4_b407_c74773470512.slice/crio-8c6800c32b72ccb99cd12d885e71de6bc3c4ea4da2ed072076ce0f4490cad896 WatchSource:0}: Error finding container 8c6800c32b72ccb99cd12d885e71de6bc3c4ea4da2ed072076ce0f4490cad896: Status 404 returned error can't find the container with id 8c6800c32b72ccb99cd12d885e71de6bc3c4ea4da2ed072076ce0f4490cad896
Nov 02 08:29:27.963949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:27.963903    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"]
Nov 02 08:29:28.040696 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:28.040664    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd" event={"ID":"2127178e-36c3-47c4-b407-c74773470512","Type":"ContainerStarted","Data":"8c6800c32b72ccb99cd12d885e71de6bc3c4ea4da2ed072076ce0f4490cad896"}
Nov 02 08:29:29.050380 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:29.050348    4898 generic.go:356] "Generic (PLEG): container finished" podID="2127178e-36c3-47c4-b407-c74773470512" containerID="3025ba2e8908bd42618de99c4ddd1a93b0d960f40ff1f0cafae1d036b8481bdb" exitCode=0
Nov 02 08:29:29.050766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:29.050438    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd" event={"ID":"2127178e-36c3-47c4-b407-c74773470512","Type":"ContainerDied","Data":"3025ba2e8908bd42618de99c4ddd1a93b0d960f40ff1f0cafae1d036b8481bdb"}
Nov 02 08:29:30.245765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:30.245738    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:30.308092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:30.308056    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2127178e-36c3-47c4-b407-c74773470512-kube-api-access-d4p6s\" (UniqueName: \"kubernetes.io/projected/2127178e-36c3-47c4-b407-c74773470512-kube-api-access-d4p6s\") pod \"2127178e-36c3-47c4-b407-c74773470512\" (UID: \"2127178e-36c3-47c4-b407-c74773470512\") "
Nov 02 08:29:30.308281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:30.308134    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2127178e-36c3-47c4-b407-c74773470512-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2127178e-36c3-47c4-b407-c74773470512-maas-api-xks-ca\") pod \"2127178e-36c3-47c4-b407-c74773470512\" (UID: \"2127178e-36c3-47c4-b407-c74773470512\") "
Nov 02 08:29:30.308281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:30.308244    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2127178e-36c3-47c4-b407-c74773470512-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2127178e-36c3-47c4-b407-c74773470512-maas-api-openshift-ca\") pod \"2127178e-36c3-47c4-b407-c74773470512\" (UID: \"2127178e-36c3-47c4-b407-c74773470512\") "
Nov 02 08:29:30.308637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:30.308597    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2127178e-36c3-47c4-b407-c74773470512-maas-api-openshift-ca" pod "2127178e-36c3-47c4-b407-c74773470512" (UID: "2127178e-36c3-47c4-b407-c74773470512"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:29:30.308932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:30.308907    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2127178e-36c3-47c4-b407-c74773470512-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:29:30.310733 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:30.310658    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2127178e-36c3-47c4-b407-c74773470512-maas-api-xks-ca" pod "2127178e-36c3-47c4-b407-c74773470512" (UID: "2127178e-36c3-47c4-b407-c74773470512"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:29:30.311093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:30.311066    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2127178e-36c3-47c4-b407-c74773470512-kube-api-access-d4p6s" pod "2127178e-36c3-47c4-b407-c74773470512" (UID: "2127178e-36c3-47c4-b407-c74773470512"). InnerVolumeSpecName "kube-api-access-d4p6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:29:30.410269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:30.410237    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4p6s\" (UniqueName: \"kubernetes.io/projected/2127178e-36c3-47c4-b407-c74773470512-kube-api-access-d4p6s\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:29:30.410269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:30.410265    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2127178e-36c3-47c4-b407-c74773470512-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:29:31.068640 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:31.068609    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"
Nov 02 08:29:31.068818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:31.068642    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd" event={"ID":"2127178e-36c3-47c4-b407-c74773470512","Type":"ContainerDied","Data":"8c6800c32b72ccb99cd12d885e71de6bc3c4ea4da2ed072076ce0f4490cad896"}
Nov 02 08:29:31.068818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:31.068696    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c6800c32b72ccb99cd12d885e71de6bc3c4ea4da2ed072076ce0f4490cad896"
Nov 02 08:29:31.443901 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:31.443859    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq" podUID="eec39c96-e0ca-439f-90be-331900d88b53" containerName="payload-processing" containerID="cri-o://e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32" gracePeriod=25
Nov 02 08:29:31.478281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:31.478235    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4" podUID="c36a913f-4117-4638-881a-c15c2938f172" containerName="payload-pre-processing" containerID="cri-o://4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20" gracePeriod=25
Nov 02 08:29:41.172499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:41.172456    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4" podUID="c36a913f-4117-4638-881a-c15c2938f172" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:29:41.173168 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:41.172570    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq" podUID="eec39c96-e0ca-439f-90be-331900d88b53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:29:44.999268 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:29:44.999226    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:29:48.048125 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:48.048084    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"]
Nov 02 08:29:48.053307 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:48.053281    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f8bc9d-8fb58"]
Nov 02 08:29:49.465758 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:49.465719    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a3ec7ac-934c-4a5b-ab71-7e63615c34b7" path="/var/lib/kubelet/pods/0a3ec7ac-934c-4a5b-ab71-7e63615c34b7/volumes"
Nov 02 08:29:51.172784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:51.172733    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4" podUID="c36a913f-4117-4638-881a-c15c2938f172" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:29:51.173319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:51.172802    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq" podUID="eec39c96-e0ca-439f-90be-331900d88b53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:29:52.121768 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.102073    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"
Nov 02 08:29:52.133127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.133092    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"
Nov 02 08:29:52.188858 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.188823    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36a913f-4117-4638-881a-c15c2938f172-kube-api-access-gzkql\" (UniqueName: \"kubernetes.io/projected/c36a913f-4117-4638-881a-c15c2938f172-kube-api-access-gzkql\") pod \"c36a913f-4117-4638-881a-c15c2938f172\" (UID: \"c36a913f-4117-4638-881a-c15c2938f172\") "
Nov 02 08:29:52.189357 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.188968    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec39c96-e0ca-439f-90be-331900d88b53-kube-api-access-xv9c6\" (UniqueName: \"kubernetes.io/projected/eec39c96-e0ca-439f-90be-331900d88b53-kube-api-access-xv9c6\") pod \"eec39c96-e0ca-439f-90be-331900d88b53\" (UID: \"eec39c96-e0ca-439f-90be-331900d88b53\") "
Nov 02 08:29:52.189357 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.189032    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c36a913f-4117-4638-881a-c15c2938f172-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c36a913f-4117-4638-881a-c15c2938f172-plugins-config-volume\") pod \"c36a913f-4117-4638-881a-c15c2938f172\" (UID: \"c36a913f-4117-4638-881a-c15c2938f172\") "
Nov 02 08:29:52.189357 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.189182    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec39c96-e0ca-439f-90be-331900d88b53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec39c96-e0ca-439f-90be-331900d88b53-plugins-config-volume\") pod \"eec39c96-e0ca-439f-90be-331900d88b53\" (UID: \"eec39c96-e0ca-439f-90be-331900d88b53\") "
Nov 02 08:29:52.189558 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.189413    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c36a913f-4117-4638-881a-c15c2938f172-plugins-config-volume" pod "c36a913f-4117-4638-881a-c15c2938f172" (UID: "c36a913f-4117-4638-881a-c15c2938f172"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:29:52.189634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.189608    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec39c96-e0ca-439f-90be-331900d88b53-plugins-config-volume" pod "eec39c96-e0ca-439f-90be-331900d88b53" (UID: "eec39c96-e0ca-439f-90be-331900d88b53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:29:52.189729 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.189709    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c36a913f-4117-4638-881a-c15c2938f172-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:29:52.191110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.191089    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36a913f-4117-4638-881a-c15c2938f172-kube-api-access-gzkql" pod "c36a913f-4117-4638-881a-c15c2938f172" (UID: "c36a913f-4117-4638-881a-c15c2938f172"). InnerVolumeSpecName "kube-api-access-gzkql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:29:52.191205 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.191175    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec39c96-e0ca-439f-90be-331900d88b53-kube-api-access-xv9c6" pod "eec39c96-e0ca-439f-90be-331900d88b53" (UID: "eec39c96-e0ca-439f-90be-331900d88b53"). InnerVolumeSpecName "kube-api-access-xv9c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:29:52.244251 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.244158    4898 generic.go:356] "Generic (PLEG): container finished" podID="eec39c96-e0ca-439f-90be-331900d88b53" containerID="e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32" exitCode=0
Nov 02 08:29:52.244251 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.244202    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"
Nov 02 08:29:52.244251 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.244231    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq" event={"ID":"eec39c96-e0ca-439f-90be-331900d88b53","Type":"ContainerDied","Data":"e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32"}
Nov 02 08:29:52.244527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.244282    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq" event={"ID":"eec39c96-e0ca-439f-90be-331900d88b53","Type":"ContainerDied","Data":"0e4fe5acf14684d63ea536e36fd31e30a7bcddc57f524dbc9d9369ce199224c8"}
Nov 02 08:29:52.244527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.244312    4898 scope.go:122] "RemoveContainer" containerID="e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32"
Nov 02 08:29:52.247006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.246015    4898 generic.go:356] "Generic (PLEG): container finished" podID="c36a913f-4117-4638-881a-c15c2938f172" containerID="4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20" exitCode=0
Nov 02 08:29:52.247006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.246145    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4" event={"ID":"c36a913f-4117-4638-881a-c15c2938f172","Type":"ContainerDied","Data":"4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20"}
Nov 02 08:29:52.247006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.246179    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4" event={"ID":"c36a913f-4117-4638-881a-c15c2938f172","Type":"ContainerDied","Data":"5e21254c498c0bf5fce8cfac966fa97e3f0c45bd67680363b32439ec259bec9d"}
Nov 02 08:29:52.247006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.246230    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"
Nov 02 08:29:52.257527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.257505    4898 scope.go:122] "RemoveContainer" containerID="e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32"
Nov 02 08:29:52.257797 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:29:52.257781    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32\": container with ID starting with e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32 not found: ID does not exist" containerID="e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32"
Nov 02 08:29:52.257850 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.257804    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32"} err="failed to get container status \"e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32\": rpc error: code = NotFound desc = could not find container \"e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32\": container with ID starting with e6e55d125a3dc9fd3d7a154b0ebd34ff73c45cc9255fa5e06d9d36460911cf32 not found: ID does not exist"
Nov 02 08:29:52.257850 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.257820    4898 scope.go:122] "RemoveContainer" containerID="4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20"
Nov 02 08:29:52.268342 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.268319    4898 scope.go:122] "RemoveContainer" containerID="4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20"
Nov 02 08:29:52.268595 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:29:52.268575    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20\": container with ID starting with 4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20 not found: ID does not exist" containerID="4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20"
Nov 02 08:29:52.268667 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.268602    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20"} err="failed to get container status \"4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20\": rpc error: code = NotFound desc = could not find container \"4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20\": container with ID starting with 4fcb7ca4cf6b0a3db48bd2e8e2ec3b698799d96a22f546cd51452f20adc2fd20 not found: ID does not exist"
Nov 02 08:29:52.290631 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.290607    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec39c96-e0ca-439f-90be-331900d88b53-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:29:52.290631 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.290629    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzkql\" (UniqueName: \"kubernetes.io/projected/c36a913f-4117-4638-881a-c15c2938f172-kube-api-access-gzkql\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:29:52.290759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.290641    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv9c6\" (UniqueName: \"kubernetes.io/projected/eec39c96-e0ca-439f-90be-331900d88b53-kube-api-access-xv9c6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:29:52.298137 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.298114    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"]
Nov 02 08:29:52.302694 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.302674    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f2e06-7d4d8cfcd7-srrv4"]
Nov 02 08:29:52.372925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.364610    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"]
Nov 02 08:29:52.384835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:52.377736    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f2e06-7dbbfc6cd9-h72zq"]
Nov 02 08:29:53.468128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:53.468091    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c36a913f-4117-4638-881a-c15c2938f172" path="/var/lib/kubelet/pods/c36a913f-4117-4638-881a-c15c2938f172/volumes"
Nov 02 08:29:53.468759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:29:53.468646    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec39c96-e0ca-439f-90be-331900d88b53" path="/var/lib/kubelet/pods/eec39c96-e0ca-439f-90be-331900d88b53/volumes"
Nov 02 08:30:00.148365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.148320    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"]
Nov 02 08:30:00.149280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149256    4898 cpu_manager.go:414] "containerMap: removing container" podUID="eec39c96-e0ca-439f-90be-331900d88b53" containerName="payload-processing"
Nov 02 08:30:00.149409 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149280    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec39c96-e0ca-439f-90be-331900d88b53" containerName="payload-processing"
Nov 02 08:30:00.149409 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149316    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c36a913f-4117-4638-881a-c15c2938f172" containerName="payload-pre-processing"
Nov 02 08:30:00.149409 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149324    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c36a913f-4117-4638-881a-c15c2938f172" containerName="payload-pre-processing"
Nov 02 08:30:00.149409 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149347    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2127178e-36c3-47c4-b407-c74773470512" containerName="revoke-keys"
Nov 02 08:30:00.149409 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149355    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2127178e-36c3-47c4-b407-c74773470512" containerName="revoke-keys"
Nov 02 08:30:00.149409 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149403    4898 cpu_manager.go:414] "containerMap: removing container" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy"
Nov 02 08:30:00.149409 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149410    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcce1599-5df4-4e0a-853a-6127ac605ffe" containerName="istio-proxy"
Nov 02 08:30:00.149785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149429    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy"
Nov 02 08:30:00.149785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149437    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9474d0bd-4733-44e8-a308-e15f2edb4108" containerName="istio-proxy"
Nov 02 08:30:00.149785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149624    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c36a913f-4117-4638-881a-c15c2938f172" containerName="payload-pre-processing"
Nov 02 08:30:00.149785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149639    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2127178e-36c3-47c4-b407-c74773470512" containerName="revoke-keys"
Nov 02 08:30:00.149785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.149661    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec39c96-e0ca-439f-90be-331900d88b53" containerName="payload-processing"
Nov 02 08:30:00.155389 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.155355    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"]
Nov 02 08:30:00.155532 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.155487    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"
Nov 02 08:30:00.161499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.161469    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"]
Nov 02 08:30:00.161651 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.161592    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"
Nov 02 08:30:00.167356 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.167332    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"]
Nov 02 08:30:00.168355 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.168331    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"
Nov 02 08:30:00.174218 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.174192    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"]
Nov 02 08:30:00.177600 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.177559    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"
Nov 02 08:30:00.187275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.185549    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:30:00.187275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.185701    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:30:00.187275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.185839    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:30:00.190127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.190103    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"]
Nov 02 08:30:00.192542 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.192091    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"
Nov 02 08:30:00.210353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.210328    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"]
Nov 02 08:30:00.212043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.210593    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"
Nov 02 08:30:00.221443 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.221095    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"]
Nov 02 08:30:00.221443 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.221153    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"]
Nov 02 08:30:00.221443 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.221229    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"
Nov 02 08:30:00.231646 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.231613    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"]
Nov 02 08:30:00.231783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.231656    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"]
Nov 02 08:30:00.231783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.231744    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"
Nov 02 08:30:00.232523 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.232493    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9zqh\" (UniqueName: \"kubernetes.io/projected/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-kube-api-access-s9zqh\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97\" (UID: \"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"
Nov 02 08:30:00.232628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.232586    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hptf9\" (UniqueName: \"kubernetes.io/projected/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-kube-api-access-hptf9\") pod \"maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t\" (UID: \"0e09d271-5894-4f0d-9a38-dca2b8efb7bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"
Nov 02 08:30:00.232698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.232632    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jx4v\" (UniqueName: \"kubernetes.io/projected/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-kube-api-access-4jx4v\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq\" (UID: \"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"
Nov 02 08:30:00.232698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.232658    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045dedc0-929e-4caf-8349-11463c81fc48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx\" (UID: \"045dedc0-929e-4caf-8349-11463c81fc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"
Nov 02 08:30:00.232698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.232683    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97\" (UID: \"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"
Nov 02 08:30:00.232869 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.232733    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t\" (UID: \"0e09d271-5894-4f0d-9a38-dca2b8efb7bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"
Nov 02 08:30:00.232869 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.232766    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvdvg\" (UniqueName: \"kubernetes.io/projected/045dedc0-929e-4caf-8349-11463c81fc48-kube-api-access-rvdvg\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx\" (UID: \"045dedc0-929e-4caf-8349-11463c81fc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"
Nov 02 08:30:00.232869 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.232787    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq\" (UID: \"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"
Nov 02 08:30:00.236169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.236128    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"]
Nov 02 08:30:00.244310 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.244287    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"]
Nov 02 08:30:00.274966 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.247237    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"]
Nov 02 08:30:00.274966 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.253365    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"]
Nov 02 08:30:00.274966 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.262967    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"
Nov 02 08:30:00.274966 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.264939    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"
Nov 02 08:30:00.293256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.289449    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"]
Nov 02 08:30:00.299319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.296445    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"]
Nov 02 08:30:00.304265 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.304100    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"
Nov 02 08:30:00.311288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.310832    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"]
Nov 02 08:30:00.317371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.317351    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"]
Nov 02 08:30:00.334638 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.334602    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0539990-8490-41d3-8dba-77f7e8a6af14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx\" (UID: \"b0539990-8490-41d3-8dba-77f7e8a6af14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"
Nov 02 08:30:00.334743 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.334657    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pdmm\" (UniqueName: \"kubernetes.io/projected/eeb4d901-47b4-4247-b099-db9bd7144ce2-kube-api-access-4pdmm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc\" (UID: \"eeb4d901-47b4-4247-b099-db9bd7144ce2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"
Nov 02 08:30:00.334743 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.334688    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7zrw\" (UniqueName: \"kubernetes.io/projected/a259b571-ddfe-400f-a22b-7f16768a82f7-kube-api-access-j7zrw\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp\" (UID: \"a259b571-ddfe-400f-a22b-7f16768a82f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"
Nov 02 08:30:00.334743 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.334723    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9dbx\" (UniqueName: \"kubernetes.io/projected/eb7a2f08-f10d-4430-b0be-60892b129043-kube-api-access-w9dbx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv\" (UID: \"eb7a2f08-f10d-4430-b0be-60892b129043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"
Nov 02 08:30:00.334873 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.334758    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9zqh\" (UniqueName: \"kubernetes.io/projected/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-kube-api-access-s9zqh\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97\" (UID: \"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"
Nov 02 08:30:00.334873 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.334803    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2abea0a3-c4f1-40df-af27-a75342569b3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74\" (UID: \"2abea0a3-c4f1-40df-af27-a75342569b3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"
Nov 02 08:30:00.334973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.334877    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hptf9\" (UniqueName: \"kubernetes.io/projected/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-kube-api-access-hptf9\") pod \"maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t\" (UID: \"0e09d271-5894-4f0d-9a38-dca2b8efb7bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"
Nov 02 08:30:00.335013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.334991    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtzt9\" (UniqueName: \"kubernetes.io/projected/2abea0a3-c4f1-40df-af27-a75342569b3b-kube-api-access-vtzt9\") pod \"maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74\" (UID: \"2abea0a3-c4f1-40df-af27-a75342569b3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"
Nov 02 08:30:00.335083 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.335067    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jx4v\" (UniqueName: \"kubernetes.io/projected/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-kube-api-access-4jx4v\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq\" (UID: \"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"
Nov 02 08:30:00.335135 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.335120    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045dedc0-929e-4caf-8349-11463c81fc48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx\" (UID: \"045dedc0-929e-4caf-8349-11463c81fc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"
Nov 02 08:30:00.335172 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.335162    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97\" (UID: \"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"
Nov 02 08:30:00.335223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.335206    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a259b571-ddfe-400f-a22b-7f16768a82f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp\" (UID: \"a259b571-ddfe-400f-a22b-7f16768a82f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"
Nov 02 08:30:00.335286 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.335274    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb7a2f08-f10d-4430-b0be-60892b129043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv\" (UID: \"eb7a2f08-f10d-4430-b0be-60892b129043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"
Nov 02 08:30:00.335322 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.335308    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t\" (UID: \"0e09d271-5894-4f0d-9a38-dca2b8efb7bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"
Nov 02 08:30:00.335362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.335351    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvdvg\" (UniqueName: \"kubernetes.io/projected/045dedc0-929e-4caf-8349-11463c81fc48-kube-api-access-rvdvg\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx\" (UID: \"045dedc0-929e-4caf-8349-11463c81fc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"
Nov 02 08:30:00.335397 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.335371    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8398fcd-e412-4abe-b630-78573f4cf7fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq\" (UID: \"d8398fcd-e412-4abe-b630-78573f4cf7fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"
Nov 02 08:30:00.335437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.335396    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq\" (UID: \"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"
Nov 02 08:30:00.335476 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.335440    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jqt\" (UniqueName: \"kubernetes.io/projected/d8398fcd-e412-4abe-b630-78573f4cf7fe-kube-api-access-g2jqt\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq\" (UID: \"d8398fcd-e412-4abe-b630-78573f4cf7fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"
Nov 02 08:30:00.335515 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.335481    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggc7c\" (UniqueName: \"kubernetes.io/projected/b0539990-8490-41d3-8dba-77f7e8a6af14-kube-api-access-ggc7c\") pod \"maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx\" (UID: \"b0539990-8490-41d3-8dba-77f7e8a6af14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"
Nov 02 08:30:00.335515 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.335497    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4d901-47b4-4247-b099-db9bd7144ce2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc\" (UID: \"eeb4d901-47b4-4247-b099-db9bd7144ce2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"
Nov 02 08:30:00.336595 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.336573    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045dedc0-929e-4caf-8349-11463c81fc48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx\" (UID: \"045dedc0-929e-4caf-8349-11463c81fc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"
Nov 02 08:30:00.337054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.337037    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97\" (UID: \"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"
Nov 02 08:30:00.337491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.337475    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t\" (UID: \"0e09d271-5894-4f0d-9a38-dca2b8efb7bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"
Nov 02 08:30:00.338117 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.338100    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq\" (UID: \"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"
Nov 02 08:30:00.338220 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.338204    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"]
Nov 02 08:30:00.341334 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.341315    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"]
Nov 02 08:30:00.348307 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.348283    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"]
Nov 02 08:30:00.348522 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.348497    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"
Nov 02 08:30:00.353784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.353762    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"]
Nov 02 08:30:00.353910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.353861    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"
Nov 02 08:30:00.358483 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.357509    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"]
Nov 02 08:30:00.360145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.360096    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9zqh\" (UniqueName: \"kubernetes.io/projected/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-kube-api-access-s9zqh\") pod \"maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97\" (UID: \"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"
Nov 02 08:30:00.360270 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.360211    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jx4v\" (UniqueName: \"kubernetes.io/projected/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-kube-api-access-4jx4v\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq\" (UID: \"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"
Nov 02 08:30:00.361879 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.361858    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hptf9\" (UniqueName: \"kubernetes.io/projected/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-kube-api-access-hptf9\") pod \"maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t\" (UID: \"0e09d271-5894-4f0d-9a38-dca2b8efb7bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"
Nov 02 08:30:00.372595 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.372569    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"
Nov 02 08:30:00.389765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.389737    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvdvg\" (UniqueName: \"kubernetes.io/projected/045dedc0-929e-4caf-8349-11463c81fc48-kube-api-access-rvdvg\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx\" (UID: \"045dedc0-929e-4caf-8349-11463c81fc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"
Nov 02 08:30:00.394955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.394922    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"]
Nov 02 08:30:00.401926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.401905    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"
Nov 02 08:30:00.402140 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.402119    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"]
Nov 02 08:30:00.407793 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.406788    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"]
Nov 02 08:30:00.411328 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.411309    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"]
Nov 02 08:30:00.422444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.422347    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"]
Nov 02 08:30:00.425985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.425961    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"]
Nov 02 08:30:00.436536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.436504    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2abea0a3-c4f1-40df-af27-a75342569b3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74\" (UID: \"2abea0a3-c4f1-40df-af27-a75342569b3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"
Nov 02 08:30:00.436634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.436562    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b72892-6169-4ffa-bc4a-e64ad46688ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq\" (UID: \"48b72892-6169-4ffa-bc4a-e64ad46688ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"
Nov 02 08:30:00.436634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.436594    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn\" (UID: \"d85b69ee-2f64-45f1-af4e-9f23b122f2c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"
Nov 02 08:30:00.436773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.436640    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtzt9\" (UniqueName: \"kubernetes.io/projected/2abea0a3-c4f1-40df-af27-a75342569b3b-kube-api-access-vtzt9\") pod \"maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74\" (UID: \"2abea0a3-c4f1-40df-af27-a75342569b3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"
Nov 02 08:30:00.436773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.436675    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8zn6\" (UniqueName: \"kubernetes.io/projected/e9d7ab63-f8af-475d-a6e8-67e7d5587603-kube-api-access-q8zn6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w\" (UID: \"e9d7ab63-f8af-475d-a6e8-67e7d5587603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"
Nov 02 08:30:00.436773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.436714    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv465\" (UniqueName: \"kubernetes.io/projected/48b72892-6169-4ffa-bc4a-e64ad46688ae-kube-api-access-cv465\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq\" (UID: \"48b72892-6169-4ffa-bc4a-e64ad46688ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"
Nov 02 08:30:00.436773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.436745    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d7ab63-f8af-475d-a6e8-67e7d5587603-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w\" (UID: \"e9d7ab63-f8af-475d-a6e8-67e7d5587603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"
Nov 02 08:30:00.436773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.436770    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfnc6\" (UniqueName: \"kubernetes.io/projected/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-kube-api-access-sfnc6\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv\" (UID: \"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"
Nov 02 08:30:00.437082 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.436835    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56th2\" (UniqueName: \"kubernetes.io/projected/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-kube-api-access-56th2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn\" (UID: \"d85b69ee-2f64-45f1-af4e-9f23b122f2c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"
Nov 02 08:30:00.437082 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.436882    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a259b571-ddfe-400f-a22b-7f16768a82f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp\" (UID: \"a259b571-ddfe-400f-a22b-7f16768a82f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"
Nov 02 08:30:00.437082 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.436996    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb7a2f08-f10d-4430-b0be-60892b129043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv\" (UID: \"eb7a2f08-f10d-4430-b0be-60892b129043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"
Nov 02 08:30:00.437655 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.437627    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb7a2f08-f10d-4430-b0be-60892b129043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv\" (UID: \"eb7a2f08-f10d-4430-b0be-60892b129043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"
Nov 02 08:30:00.437655 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.437639    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a259b571-ddfe-400f-a22b-7f16768a82f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp\" (UID: \"a259b571-ddfe-400f-a22b-7f16768a82f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"
Nov 02 08:30:00.437866 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.437840    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv\" (UID: \"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"
Nov 02 08:30:00.437982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.437929    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8398fcd-e412-4abe-b630-78573f4cf7fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq\" (UID: \"d8398fcd-e412-4abe-b630-78573f4cf7fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"
Nov 02 08:30:00.438098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.438079    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2abea0a3-c4f1-40df-af27-a75342569b3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74\" (UID: \"2abea0a3-c4f1-40df-af27-a75342569b3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"
Nov 02 08:30:00.438259 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.438236    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jqt\" (UniqueName: \"kubernetes.io/projected/d8398fcd-e412-4abe-b630-78573f4cf7fe-kube-api-access-g2jqt\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq\" (UID: \"d8398fcd-e412-4abe-b630-78573f4cf7fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"
Nov 02 08:30:00.438324 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.438308    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggc7c\" (UniqueName: \"kubernetes.io/projected/b0539990-8490-41d3-8dba-77f7e8a6af14-kube-api-access-ggc7c\") pod \"maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx\" (UID: \"b0539990-8490-41d3-8dba-77f7e8a6af14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"
Nov 02 08:30:00.438378 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.438338    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4d901-47b4-4247-b099-db9bd7144ce2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc\" (UID: \"eeb4d901-47b4-4247-b099-db9bd7144ce2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"
Nov 02 08:30:00.438440 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.438387    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0539990-8490-41d3-8dba-77f7e8a6af14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx\" (UID: \"b0539990-8490-41d3-8dba-77f7e8a6af14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"
Nov 02 08:30:00.438440 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.438414    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdmm\" (UniqueName: \"kubernetes.io/projected/eeb4d901-47b4-4247-b099-db9bd7144ce2-kube-api-access-4pdmm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc\" (UID: \"eeb4d901-47b4-4247-b099-db9bd7144ce2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"
Nov 02 08:30:00.438549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.438441    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7zrw\" (UniqueName: \"kubernetes.io/projected/a259b571-ddfe-400f-a22b-7f16768a82f7-kube-api-access-j7zrw\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp\" (UID: \"a259b571-ddfe-400f-a22b-7f16768a82f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"
Nov 02 08:30:00.438549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.438473    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9dbx\" (UniqueName: \"kubernetes.io/projected/eb7a2f08-f10d-4430-b0be-60892b129043-kube-api-access-w9dbx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv\" (UID: \"eb7a2f08-f10d-4430-b0be-60892b129043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"
Nov 02 08:30:00.438734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.438709    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8398fcd-e412-4abe-b630-78573f4cf7fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq\" (UID: \"d8398fcd-e412-4abe-b630-78573f4cf7fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"
Nov 02 08:30:00.439155 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.439129    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4d901-47b4-4247-b099-db9bd7144ce2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc\" (UID: \"eeb4d901-47b4-4247-b099-db9bd7144ce2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"
Nov 02 08:30:00.439534 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.439510    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0539990-8490-41d3-8dba-77f7e8a6af14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx\" (UID: \"b0539990-8490-41d3-8dba-77f7e8a6af14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"
Nov 02 08:30:00.460038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.460011    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtzt9\" (UniqueName: \"kubernetes.io/projected/2abea0a3-c4f1-40df-af27-a75342569b3b-kube-api-access-vtzt9\") pod \"maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74\" (UID: \"2abea0a3-c4f1-40df-af27-a75342569b3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"
Nov 02 08:30:00.462654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.462588    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jqt\" (UniqueName: \"kubernetes.io/projected/d8398fcd-e412-4abe-b630-78573f4cf7fe-kube-api-access-g2jqt\") pod \"maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq\" (UID: \"d8398fcd-e412-4abe-b630-78573f4cf7fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"
Nov 02 08:30:00.464988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.464940    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9dbx\" (UniqueName: \"kubernetes.io/projected/eb7a2f08-f10d-4430-b0be-60892b129043-kube-api-access-w9dbx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv\" (UID: \"eb7a2f08-f10d-4430-b0be-60892b129043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"
Nov 02 08:30:00.465100 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.465084    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7zrw\" (UniqueName: \"kubernetes.io/projected/a259b571-ddfe-400f-a22b-7f16768a82f7-kube-api-access-j7zrw\") pod \"maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp\" (UID: \"a259b571-ddfe-400f-a22b-7f16768a82f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"
Nov 02 08:30:00.465581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.465559    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggc7c\" (UniqueName: \"kubernetes.io/projected/b0539990-8490-41d3-8dba-77f7e8a6af14-kube-api-access-ggc7c\") pod \"maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx\" (UID: \"b0539990-8490-41d3-8dba-77f7e8a6af14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"
Nov 02 08:30:00.465634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.465559    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pdmm\" (UniqueName: \"kubernetes.io/projected/eeb4d901-47b4-4247-b099-db9bd7144ce2-kube-api-access-4pdmm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc\" (UID: \"eeb4d901-47b4-4247-b099-db9bd7144ce2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"
Nov 02 08:30:00.490748 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.490726    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"
Nov 02 08:30:00.509298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.509271    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"
Nov 02 08:30:00.530014 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.529979    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"
Nov 02 08:30:00.540067 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.540039    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d7ab63-f8af-475d-a6e8-67e7d5587603-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w\" (UID: \"e9d7ab63-f8af-475d-a6e8-67e7d5587603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"
Nov 02 08:30:00.540215 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.540081    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfnc6\" (UniqueName: \"kubernetes.io/projected/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-kube-api-access-sfnc6\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv\" (UID: \"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"
Nov 02 08:30:00.540215 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.540138    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzwn\" (UniqueName: \"kubernetes.io/projected/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-kube-api-access-xrzwn\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw\" (UID: \"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"
Nov 02 08:30:00.540215 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.540187    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56th2\" (UniqueName: \"kubernetes.io/projected/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-kube-api-access-56th2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn\" (UID: \"d85b69ee-2f64-45f1-af4e-9f23b122f2c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"
Nov 02 08:30:00.540398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.540312    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv\" (UID: \"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"
Nov 02 08:30:00.540516 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.540499    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw\" (UID: \"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"
Nov 02 08:30:00.540602 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.540582    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b72892-6169-4ffa-bc4a-e64ad46688ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq\" (UID: \"48b72892-6169-4ffa-bc4a-e64ad46688ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"
Nov 02 08:30:00.540662 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.540616    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn\" (UID: \"d85b69ee-2f64-45f1-af4e-9f23b122f2c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"
Nov 02 08:30:00.540771 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.540693    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8zn6\" (UniqueName: \"kubernetes.io/projected/e9d7ab63-f8af-475d-a6e8-67e7d5587603-kube-api-access-q8zn6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w\" (UID: \"e9d7ab63-f8af-475d-a6e8-67e7d5587603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"
Nov 02 08:30:00.540771 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.540736    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv465\" (UniqueName: \"kubernetes.io/projected/48b72892-6169-4ffa-bc4a-e64ad46688ae-kube-api-access-cv465\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq\" (UID: \"48b72892-6169-4ffa-bc4a-e64ad46688ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"
Nov 02 08:30:00.540771 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.540756    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d7ab63-f8af-475d-a6e8-67e7d5587603-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w\" (UID: \"e9d7ab63-f8af-475d-a6e8-67e7d5587603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"
Nov 02 08:30:00.541566 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.541543    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv\" (UID: \"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"
Nov 02 08:30:00.541682 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.541579    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn\" (UID: \"d85b69ee-2f64-45f1-af4e-9f23b122f2c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"
Nov 02 08:30:00.541875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.541852    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b72892-6169-4ffa-bc4a-e64ad46688ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq\" (UID: \"48b72892-6169-4ffa-bc4a-e64ad46688ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"
Nov 02 08:30:00.571704 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.570454    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfnc6\" (UniqueName: \"kubernetes.io/projected/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-kube-api-access-sfnc6\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv\" (UID: \"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"
Nov 02 08:30:00.571704 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.570712    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"
Nov 02 08:30:00.572673 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.572618    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56th2\" (UniqueName: \"kubernetes.io/projected/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-kube-api-access-56th2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn\" (UID: \"d85b69ee-2f64-45f1-af4e-9f23b122f2c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"
Nov 02 08:30:00.576305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.575543    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8zn6\" (UniqueName: \"kubernetes.io/projected/e9d7ab63-f8af-475d-a6e8-67e7d5587603-kube-api-access-q8zn6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w\" (UID: \"e9d7ab63-f8af-475d-a6e8-67e7d5587603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"
Nov 02 08:30:00.579416 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.579105    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"
Nov 02 08:30:00.584565 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.584311    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv465\" (UniqueName: \"kubernetes.io/projected/48b72892-6169-4ffa-bc4a-e64ad46688ae-kube-api-access-cv465\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq\" (UID: \"48b72892-6169-4ffa-bc4a-e64ad46688ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"
Nov 02 08:30:00.596959 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.596738    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"
Nov 02 08:30:00.610756 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.610525    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"
Nov 02 08:30:00.618377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.618266    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"
Nov 02 08:30:00.635421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.635203    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"
Nov 02 08:30:00.645559 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.641933    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw\" (UID: \"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"
Nov 02 08:30:00.645559 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.642082    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzwn\" (UniqueName: \"kubernetes.io/projected/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-kube-api-access-xrzwn\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw\" (UID: \"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"
Nov 02 08:30:00.645559 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.643423    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw\" (UID: \"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"
Nov 02 08:30:00.650145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.650087    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"
Nov 02 08:30:00.660657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.660511    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"
Nov 02 08:30:00.671719 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.671688    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzwn\" (UniqueName: \"kubernetes.io/projected/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-kube-api-access-xrzwn\") pod \"maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw\" (UID: \"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"
Nov 02 08:30:00.710409 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.710174    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"
Nov 02 08:30:00.722095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.721838    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"
Nov 02 08:30:00.730923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.730692    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"
Nov 02 08:30:00.743920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.743340    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"
Nov 02 08:30:00.870256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:00.865243    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"]
Nov 02 08:30:01.469303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:01.461538    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"]
Nov 02 08:30:01.475581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:01.475541    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"]
Nov 02 08:30:01.731661 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:01.708156    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97" event={"ID":"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc","Type":"ContainerStarted","Data":"7a4037d23994126d9c9ae5ee43adcd378e59dc19d0a47679279d0468def99f66"}
Nov 02 08:30:01.762377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:01.762327    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t" event={"ID":"0e09d271-5894-4f0d-9a38-dca2b8efb7bb","Type":"ContainerStarted","Data":"3449c09075ce18744caa6da534f04240023a98b4d5fc4b62747f9ef39bc441ec"}
Nov 02 08:30:01.955839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:01.955739    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"]
Nov 02 08:30:01.956342 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:01.956291    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy" containerID="cri-o://f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b" gracePeriod=30
Nov 02 08:30:03.016129 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.016094    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"]
Nov 02 08:30:03.338971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.338056    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"]
Nov 02 08:30:03.349962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.349929    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"]
Nov 02 08:30:03.452049 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.446988    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"]
Nov 02 08:30:03.633376 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.633186    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx" event={"ID":"b0539990-8490-41d3-8dba-77f7e8a6af14","Type":"ContainerStarted","Data":"0258898f681d6169d7cf6a015dd58fd0e3c158d50b2c07f8c7e686a13069c661"}
Nov 02 08:30:03.633376 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.633269    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx" event={"ID":"045dedc0-929e-4caf-8349-11463c81fc48","Type":"ContainerStarted","Data":"ec5ea67753b1cea7097ac803a05e4196be2444b537b35b45a13a9d04d6a309b1"}
Nov 02 08:30:03.633376 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.633323    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"]
Nov 02 08:30:03.633376 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.633363    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"]
Nov 02 08:30:03.633628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.633408    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv" event={"ID":"eb7a2f08-f10d-4430-b0be-60892b129043","Type":"ContainerStarted","Data":"da2c33e8b34fa35afe7a6fa36517c9e2b716526c2261bd094aa8474e111db515"}
Nov 02 08:30:03.633628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.633441    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"]
Nov 02 08:30:03.633628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.633485    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn" event={"ID":"d85b69ee-2f64-45f1-af4e-9f23b122f2c8","Type":"ContainerStarted","Data":"b4b48b5595a98cfe8e1cccfc11f23bd22e83716f58406e533a8cd963801c189d"}
Nov 02 08:30:03.639089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.638981    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"]
Nov 02 08:30:03.684001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.683969    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"]
Nov 02 08:30:03.684240 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.684224    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"]
Nov 02 08:30:03.685376 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.685351    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"]
Nov 02 08:30:03.686089 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.686059    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w" event={"ID":"e9d7ab63-f8af-475d-a6e8-67e7d5587603","Type":"ContainerStarted","Data":"cbc8fca8f96cdefac671d0137e4a6a84c4e06112dd45e0affd46dd688ac08751"}
Nov 02 08:30:03.691182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.691156    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq" event={"ID":"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab","Type":"ContainerStarted","Data":"81b3a6111733496420ee3866c586e5df3b8634e0e7f5b185eb7731da0bbecb9e"}
Nov 02 08:30:03.692166 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.692134    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq" event={"ID":"48b72892-6169-4ffa-bc4a-e64ad46688ae","Type":"ContainerStarted","Data":"51571d10769cc1a13bb7392e123b45f42c953f04d326c8a40a5aba24cd29e838"}
Nov 02 08:30:03.695899 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.695803    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv" event={"ID":"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f","Type":"ContainerStarted","Data":"c92808ad21eb3fac6dae4a953354e0ff0e277040945133b19886a87331f688fc"}
Nov 02 08:30:03.700213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.700184    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74" event={"ID":"2abea0a3-c4f1-40df-af27-a75342569b3b","Type":"ContainerStarted","Data":"d51f3fefe529ba01e7d05ca835baeb32aa7e6ec3fe39c7fb291902755f8b7414"}
Nov 02 08:30:03.702092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.702072    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"]
Nov 02 08:30:03.703488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.703464    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp" event={"ID":"a259b571-ddfe-400f-a22b-7f16768a82f7","Type":"ContainerStarted","Data":"b3ca00741947492cf04cd1fca693c51c3929dbb920dd2eb8053983dc5caf2aeb"}
Nov 02 08:30:03.703985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.703727    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx" podStartSLOduration=3.703709307 podStartE2EDuration="3.703709307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:03.637022157 +0000 UTC m=+3954.327349122" watchObservedRunningTime="2026-11-02 08:30:03.703709307 +0000 UTC m=+3954.394036275"
Nov 02 08:30:03.707472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.707430    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq" event={"ID":"d8398fcd-e412-4abe-b630-78573f4cf7fe","Type":"ContainerStarted","Data":"66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33"}
Nov 02 08:30:03.711701 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.711653    4898 generic.go:356] "Generic (PLEG): container finished" podID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" containerID="b43121a16a72f7361618e6cae34a3d96ace34764edd06bed02a711bc21a17cf0" exitCode=60
Nov 02 08:30:03.712764 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.712743    4898 scope.go:122] "RemoveContainer" containerID="b43121a16a72f7361618e6cae34a3d96ace34764edd06bed02a711bc21a17cf0"
Nov 02 08:30:03.716662 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.712922    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97" event={"ID":"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc","Type":"ContainerDied","Data":"b43121a16a72f7361618e6cae34a3d96ace34764edd06bed02a711bc21a17cf0"}
Nov 02 08:30:03.719540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.719513    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t" event={"ID":"0e09d271-5894-4f0d-9a38-dca2b8efb7bb","Type":"ContainerStarted","Data":"c6096d223d009868d74103e756f670a342392cde76a4c0ce453cf8db665ab005"}
Nov 02 08:30:03.743317 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:03.739709    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3109b9e6_5dfe_4422_a630_b9c42c6bc1ab.slice/crio-81b3a6111733496420ee3866c586e5df3b8634e0e7f5b185eb7731da0bbecb9e\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:03.774425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.774287    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc" event={"ID":"eeb4d901-47b4-4247-b099-db9bd7144ce2","Type":"ContainerStarted","Data":"11dda57575111a1ffbf561ab334751cf384c48d9552b695935adbe02c641ae7e"}
Nov 02 08:30:03.813723 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:03.813555    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t" podStartSLOduration=3.813536363 podStartE2EDuration="3.813536363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:03.811119947 +0000 UTC m=+3954.501446912" watchObservedRunningTime="2026-11-02 08:30:03.813536363 +0000 UTC m=+3954.503863328"
Nov 02 08:30:04.789720 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.789688    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv" event={"ID":"eb7a2f08-f10d-4430-b0be-60892b129043","Type":"ContainerStarted","Data":"9167a25cbbbfd1f6585da5150080b6c233fb46875cac0e507deb2164679afbe5"}
Nov 02 08:30:04.791598 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.791567    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn" event={"ID":"d85b69ee-2f64-45f1-af4e-9f23b122f2c8","Type":"ContainerStarted","Data":"dcf1e2ae0398f0ee0b206faf73711168d46b6dbe27b3de8c8fea0c9f72216b7c"}
Nov 02 08:30:04.793781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.793686    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w" event={"ID":"e9d7ab63-f8af-475d-a6e8-67e7d5587603","Type":"ContainerStarted","Data":"666fab951ecc4db38fe68fcaed690c39a64a2d1ed068cdb847d15428f1c82002"}
Nov 02 08:30:04.796010 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.795980    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq" event={"ID":"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab","Type":"ContainerStarted","Data":"11a412179060025e30e1818f6d9c6eb5fdd5ea5cc3b94e9102a84774fc03864e"}
Nov 02 08:30:04.797783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.797753    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq" event={"ID":"48b72892-6169-4ffa-bc4a-e64ad46688ae","Type":"ContainerStarted","Data":"81e2fcd619836f780d0b1e3a198470d2b2e572e9e1a32d0fee6ec8e226ad89d3"}
Nov 02 08:30:04.799518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.799488    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv" event={"ID":"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f","Type":"ContainerStarted","Data":"2ad84540c2e1b9493b629b4daa4aa051796992ec8bf0d8213ca1b5d61287df87"}
Nov 02 08:30:04.801141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.801117    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74" event={"ID":"2abea0a3-c4f1-40df-af27-a75342569b3b","Type":"ContainerStarted","Data":"d328dd5f1b39cf0de4bd17dbeaf4a441b92ad5acc4807c37ab42ae43f3761d36"}
Nov 02 08:30:04.802798 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.802777    4898 generic.go:356] "Generic (PLEG): container finished" podID="a259b571-ddfe-400f-a22b-7f16768a82f7" containerID="b4a74dfab5b7a312e7626b687d04e3d31a3d06dae8f8fbc1dd0aa0d5f8e448c6" exitCode=60
Nov 02 08:30:04.802932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.802838    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp" event={"ID":"a259b571-ddfe-400f-a22b-7f16768a82f7","Type":"ContainerDied","Data":"b4a74dfab5b7a312e7626b687d04e3d31a3d06dae8f8fbc1dd0aa0d5f8e448c6"}
Nov 02 08:30:04.803037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.803019    4898 scope.go:122] "RemoveContainer" containerID="b4a74dfab5b7a312e7626b687d04e3d31a3d06dae8f8fbc1dd0aa0d5f8e448c6"
Nov 02 08:30:04.805600 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.805577    4898 generic.go:356] "Generic (PLEG): container finished" podID="d8398fcd-e412-4abe-b630-78573f4cf7fe" containerID="2356b15d53e3971669bcdbcc95490114dc7cafa874a9c474ee160916a304755d" exitCode=60
Nov 02 08:30:04.805685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.805654    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq" event={"ID":"d8398fcd-e412-4abe-b630-78573f4cf7fe","Type":"ContainerDied","Data":"2356b15d53e3971669bcdbcc95490114dc7cafa874a9c474ee160916a304755d"}
Nov 02 08:30:04.805901 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.805877    4898 scope.go:122] "RemoveContainer" containerID="2356b15d53e3971669bcdbcc95490114dc7cafa874a9c474ee160916a304755d"
Nov 02 08:30:04.807956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.807929    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97" event={"ID":"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc","Type":"ContainerStarted","Data":"f268d3c646cb58801872f0f2f70252e3dc34cff04021e111fe3e8aa6f23f0f12"}
Nov 02 08:30:04.810006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.809982    4898 generic.go:356] "Generic (PLEG): container finished" podID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" containerID="c6096d223d009868d74103e756f670a342392cde76a4c0ce453cf8db665ab005" exitCode=60
Nov 02 08:30:04.810107 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.810014    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t" event={"ID":"0e09d271-5894-4f0d-9a38-dca2b8efb7bb","Type":"ContainerDied","Data":"c6096d223d009868d74103e756f670a342392cde76a4c0ce453cf8db665ab005"}
Nov 02 08:30:04.810305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.810287    4898 scope.go:122] "RemoveContainer" containerID="c6096d223d009868d74103e756f670a342392cde76a4c0ce453cf8db665ab005"
Nov 02 08:30:04.812032 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.812005    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc" event={"ID":"eeb4d901-47b4-4247-b099-db9bd7144ce2","Type":"ContainerStarted","Data":"77ce45b52ce89b980aacafcd05bf72972a9e9364a18515013af86f69726fda7d"}
Nov 02 08:30:04.813932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.813907    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx" event={"ID":"b0539990-8490-41d3-8dba-77f7e8a6af14","Type":"ContainerStarted","Data":"12f73efd8ab1b08688312f4e2c3b4920d2525ccd76a9986e9a8b4bd9d33c2eaa"}
Nov 02 08:30:04.815947 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.815920    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw" event={"ID":"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32","Type":"ContainerStarted","Data":"517c3d339113203ea1efc52bd062c9c9e0dc70b5fc7cf284edf6aa89bb0370a4"}
Nov 02 08:30:04.816060 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.815971    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw" event={"ID":"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32","Type":"ContainerStarted","Data":"17f7cd13d1ed60f766a6959927dce2adab4269555a221d1ae2202d955c92a290"}
Nov 02 08:30:04.818562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.818535    4898 generic.go:356] "Generic (PLEG): container finished" podID="045dedc0-929e-4caf-8349-11463c81fc48" containerID="b6d6bc8e4fe80ae6ec3702882d888be8f03f8122755722de3859f76c3d2127b3" exitCode=60
Nov 02 08:30:04.818562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.818574    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx" event={"ID":"045dedc0-929e-4caf-8349-11463c81fc48","Type":"ContainerDied","Data":"b6d6bc8e4fe80ae6ec3702882d888be8f03f8122755722de3859f76c3d2127b3"}
Nov 02 08:30:04.819021 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.818881    4898 scope.go:122] "RemoveContainer" containerID="b6d6bc8e4fe80ae6ec3702882d888be8f03f8122755722de3859f76c3d2127b3"
Nov 02 08:30:04.830271 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.830218    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv" podStartSLOduration=4.830199357 podStartE2EDuration="4.830199357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:04.82969296 +0000 UTC m=+3955.520019927" watchObservedRunningTime="2026-11-02 08:30:04.830199357 +0000 UTC m=+3955.520526328"
Nov 02 08:30:04.879018 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.877986    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97" podStartSLOduration=4.877972004 podStartE2EDuration="4.877972004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:04.874242857 +0000 UTC m=+3955.564569822" watchObservedRunningTime="2026-11-02 08:30:04.877972004 +0000 UTC m=+3955.568298964"
Nov 02 08:30:04.918652 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:04.918597    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc" podStartSLOduration=4.918583056 podStartE2EDuration="4.918583056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:04.913704695 +0000 UTC m=+3955.604031662" watchObservedRunningTime="2026-11-02 08:30:04.918583056 +0000 UTC m=+3955.608910066"
Nov 02 08:30:05.003861 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.003791    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx" podStartSLOduration=5.003771724 podStartE2EDuration="5.003771724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:05.003077252 +0000 UTC m=+3955.693404264" watchObservedRunningTime="2026-11-02 08:30:05.003771724 +0000 UTC m=+3955.694098690"
Nov 02 08:30:05.045444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.045324    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn" podStartSLOduration=5.045308887 podStartE2EDuration="5.045308887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:05.042942902 +0000 UTC m=+3955.733269891" watchObservedRunningTime="2026-11-02 08:30:05.045308887 +0000 UTC m=+3955.735635895"
Nov 02 08:30:05.131776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.131723    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq" podStartSLOduration=5.131707678 podStartE2EDuration="5.131707678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:05.126934659 +0000 UTC m=+3955.817261626" watchObservedRunningTime="2026-11-02 08:30:05.131707678 +0000 UTC m=+3955.822034658"
Nov 02 08:30:05.168313 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.168264    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74" podStartSLOduration=5.168249929 podStartE2EDuration="5.168249929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:05.166137648 +0000 UTC m=+3955.856464639" watchObservedRunningTime="2026-11-02 08:30:05.168249929 +0000 UTC m=+3955.858576925"
Nov 02 08:30:05.207243 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.207187    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq" podStartSLOduration=5.207171507 podStartE2EDuration="5.207171507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:05.205186061 +0000 UTC m=+3955.895513025" watchObservedRunningTime="2026-11-02 08:30:05.207171507 +0000 UTC m=+3955.897498507"
Nov 02 08:30:05.248982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.248927    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv" podStartSLOduration=5.248913076 podStartE2EDuration="5.248913076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:05.244499563 +0000 UTC m=+3955.934826528" watchObservedRunningTime="2026-11-02 08:30:05.248913076 +0000 UTC m=+3955.939240040"
Nov 02 08:30:05.287187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.287125    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw" podStartSLOduration=5.287108127 podStartE2EDuration="5.287108127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:05.283499544 +0000 UTC m=+3955.973826507" watchObservedRunningTime="2026-11-02 08:30:05.287108127 +0000 UTC m=+3955.977435133"
Nov 02 08:30:05.406749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.406699    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w" podStartSLOduration=5.406684566 podStartE2EDuration="5.406684566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:05.40271144 +0000 UTC m=+3956.093038431" watchObservedRunningTime="2026-11-02 08:30:05.406684566 +0000 UTC m=+3956.097011547"
Nov 02 08:30:05.845168 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.845128    4898 generic.go:356] "Generic (PLEG): container finished" podID="eeb4d901-47b4-4247-b099-db9bd7144ce2" containerID="77ce45b52ce89b980aacafcd05bf72972a9e9364a18515013af86f69726fda7d" exitCode=60
Nov 02 08:30:05.845661 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.845213    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc" event={"ID":"eeb4d901-47b4-4247-b099-db9bd7144ce2","Type":"ContainerDied","Data":"77ce45b52ce89b980aacafcd05bf72972a9e9364a18515013af86f69726fda7d"}
Nov 02 08:30:05.845661 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.845497    4898 scope.go:122] "RemoveContainer" containerID="77ce45b52ce89b980aacafcd05bf72972a9e9364a18515013af86f69726fda7d"
Nov 02 08:30:05.847437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.847416    4898 generic.go:356] "Generic (PLEG): container finished" podID="b0539990-8490-41d3-8dba-77f7e8a6af14" containerID="12f73efd8ab1b08688312f4e2c3b4920d2525ccd76a9986e9a8b4bd9d33c2eaa" exitCode=60
Nov 02 08:30:05.847519 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.847495    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx" event={"ID":"b0539990-8490-41d3-8dba-77f7e8a6af14","Type":"ContainerDied","Data":"12f73efd8ab1b08688312f4e2c3b4920d2525ccd76a9986e9a8b4bd9d33c2eaa"}
Nov 02 08:30:05.847736 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.847716    4898 scope.go:122] "RemoveContainer" containerID="12f73efd8ab1b08688312f4e2c3b4920d2525ccd76a9986e9a8b4bd9d33c2eaa"
Nov 02 08:30:05.850256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.850233    4898 generic.go:356] "Generic (PLEG): container finished" podID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" containerID="517c3d339113203ea1efc52bd062c9c9e0dc70b5fc7cf284edf6aa89bb0370a4" exitCode=60
Nov 02 08:30:05.850353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.850318    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw" event={"ID":"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32","Type":"ContainerDied","Data":"517c3d339113203ea1efc52bd062c9c9e0dc70b5fc7cf284edf6aa89bb0370a4"}
Nov 02 08:30:05.850498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.850484    4898 scope.go:122] "RemoveContainer" containerID="517c3d339113203ea1efc52bd062c9c9e0dc70b5fc7cf284edf6aa89bb0370a4"
Nov 02 08:30:05.852335 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.852313    4898 generic.go:356] "Generic (PLEG): container finished" podID="eb7a2f08-f10d-4430-b0be-60892b129043" containerID="9167a25cbbbfd1f6585da5150080b6c233fb46875cac0e507deb2164679afbe5" exitCode=60
Nov 02 08:30:05.852421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.852396    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv" event={"ID":"eb7a2f08-f10d-4430-b0be-60892b129043","Type":"ContainerDied","Data":"9167a25cbbbfd1f6585da5150080b6c233fb46875cac0e507deb2164679afbe5"}
Nov 02 08:30:05.852623 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.852605    4898 scope.go:122] "RemoveContainer" containerID="9167a25cbbbfd1f6585da5150080b6c233fb46875cac0e507deb2164679afbe5"
Nov 02 08:30:05.854403 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.854382    4898 generic.go:356] "Generic (PLEG): container finished" podID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" containerID="dcf1e2ae0398f0ee0b206faf73711168d46b6dbe27b3de8c8fea0c9f72216b7c" exitCode=60
Nov 02 08:30:05.854604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.854458    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn" event={"ID":"d85b69ee-2f64-45f1-af4e-9f23b122f2c8","Type":"ContainerDied","Data":"dcf1e2ae0398f0ee0b206faf73711168d46b6dbe27b3de8c8fea0c9f72216b7c"}
Nov 02 08:30:05.854871 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.854667    4898 scope.go:122] "RemoveContainer" containerID="dcf1e2ae0398f0ee0b206faf73711168d46b6dbe27b3de8c8fea0c9f72216b7c"
Nov 02 08:30:05.856477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.856453    4898 generic.go:356] "Generic (PLEG): container finished" podID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" containerID="666fab951ecc4db38fe68fcaed690c39a64a2d1ed068cdb847d15428f1c82002" exitCode=60
Nov 02 08:30:05.856591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.856570    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w" event={"ID":"e9d7ab63-f8af-475d-a6e8-67e7d5587603","Type":"ContainerDied","Data":"666fab951ecc4db38fe68fcaed690c39a64a2d1ed068cdb847d15428f1c82002"}
Nov 02 08:30:05.856824 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.856807    4898 scope.go:122] "RemoveContainer" containerID="666fab951ecc4db38fe68fcaed690c39a64a2d1ed068cdb847d15428f1c82002"
Nov 02 08:30:05.859043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.859024    4898 generic.go:356] "Generic (PLEG): container finished" podID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" containerID="11a412179060025e30e1818f6d9c6eb5fdd5ea5cc3b94e9102a84774fc03864e" exitCode=60
Nov 02 08:30:05.859693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.859670    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq" event={"ID":"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab","Type":"ContainerDied","Data":"11a412179060025e30e1818f6d9c6eb5fdd5ea5cc3b94e9102a84774fc03864e"}
Nov 02 08:30:05.860105 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.860086    4898 scope.go:122] "RemoveContainer" containerID="11a412179060025e30e1818f6d9c6eb5fdd5ea5cc3b94e9102a84774fc03864e"
Nov 02 08:30:05.863445 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.863421    4898 generic.go:356] "Generic (PLEG): container finished" podID="48b72892-6169-4ffa-bc4a-e64ad46688ae" containerID="81e2fcd619836f780d0b1e3a198470d2b2e572e9e1a32d0fee6ec8e226ad89d3" exitCode=60
Nov 02 08:30:05.863540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.863504    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq" event={"ID":"48b72892-6169-4ffa-bc4a-e64ad46688ae","Type":"ContainerDied","Data":"81e2fcd619836f780d0b1e3a198470d2b2e572e9e1a32d0fee6ec8e226ad89d3"}
Nov 02 08:30:05.863686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.863671    4898 scope.go:122] "RemoveContainer" containerID="81e2fcd619836f780d0b1e3a198470d2b2e572e9e1a32d0fee6ec8e226ad89d3"
Nov 02 08:30:05.865380 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.865356    4898 generic.go:356] "Generic (PLEG): container finished" podID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" containerID="2ad84540c2e1b9493b629b4daa4aa051796992ec8bf0d8213ca1b5d61287df87" exitCode=60
Nov 02 08:30:05.865471 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.865439    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv" event={"ID":"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f","Type":"ContainerDied","Data":"2ad84540c2e1b9493b629b4daa4aa051796992ec8bf0d8213ca1b5d61287df87"}
Nov 02 08:30:05.865658 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.865641    4898 scope.go:122] "RemoveContainer" containerID="2ad84540c2e1b9493b629b4daa4aa051796992ec8bf0d8213ca1b5d61287df87"
Nov 02 08:30:05.867289 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.867270    4898 generic.go:356] "Generic (PLEG): container finished" podID="2abea0a3-c4f1-40df-af27-a75342569b3b" containerID="d328dd5f1b39cf0de4bd17dbeaf4a441b92ad5acc4807c37ab42ae43f3761d36" exitCode=60
Nov 02 08:30:05.867410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.867331    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74" event={"ID":"2abea0a3-c4f1-40df-af27-a75342569b3b","Type":"ContainerDied","Data":"d328dd5f1b39cf0de4bd17dbeaf4a441b92ad5acc4807c37ab42ae43f3761d36"}
Nov 02 08:30:05.867587 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.867571    4898 scope.go:122] "RemoveContainer" containerID="d328dd5f1b39cf0de4bd17dbeaf4a441b92ad5acc4807c37ab42ae43f3761d36"
Nov 02 08:30:05.869524 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.869502    4898 generic.go:356] "Generic (PLEG): container finished" podID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" containerID="f268d3c646cb58801872f0f2f70252e3dc34cff04021e111fe3e8aa6f23f0f12" exitCode=60
Nov 02 08:30:05.869613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.869552    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97" event={"ID":"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc","Type":"ContainerDied","Data":"f268d3c646cb58801872f0f2f70252e3dc34cff04021e111fe3e8aa6f23f0f12"}
Nov 02 08:30:05.869613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.869599    4898 scope.go:122] "RemoveContainer" containerID="b43121a16a72f7361618e6cae34a3d96ace34764edd06bed02a711bc21a17cf0"
Nov 02 08:30:05.869836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:05.869817    4898 scope.go:122] "RemoveContainer" containerID="f268d3c646cb58801872f0f2f70252e3dc34cff04021e111fe3e8aa6f23f0f12"
Nov 02 08:30:05.870323 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:05.870285    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97_odh-ai-gateway-infra(f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc"
Nov 02 08:30:06.882743 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.882663    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv" event={"ID":"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f","Type":"ContainerStarted","Data":"2ead689a5e2d48b4786d122736a6f1060764f24fb71bedba2408d0f965bf9341"}
Nov 02 08:30:06.884899 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.884862    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74" event={"ID":"2abea0a3-c4f1-40df-af27-a75342569b3b","Type":"ContainerStarted","Data":"04864cbe72c260a18b98aa83e2f852cb841722c5b34010b65425dc680f92989f"}
Nov 02 08:30:06.886964 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.886939    4898 generic.go:356] "Generic (PLEG): container finished" podID="a259b571-ddfe-400f-a22b-7f16768a82f7" containerID="4744a29e1cb5b93578cc071c8c3be91bcc1b6b712a48a4c81c46bf5fcc2a95e6" exitCode=60
Nov 02 08:30:06.887095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.887015    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp" event={"ID":"a259b571-ddfe-400f-a22b-7f16768a82f7","Type":"ContainerDied","Data":"4744a29e1cb5b93578cc071c8c3be91bcc1b6b712a48a4c81c46bf5fcc2a95e6"}
Nov 02 08:30:06.887095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.887080    4898 scope.go:122] "RemoveContainer" containerID="b4a74dfab5b7a312e7626b687d04e3d31a3d06dae8f8fbc1dd0aa0d5f8e448c6"
Nov 02 08:30:06.887319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.887296    4898 scope.go:122] "RemoveContainer" containerID="4744a29e1cb5b93578cc071c8c3be91bcc1b6b712a48a4c81c46bf5fcc2a95e6"
Nov 02 08:30:06.887661 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:06.887617    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp_odh-ai-gateway-infra(a259b571-ddfe-400f-a22b-7f16768a82f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7"
Nov 02 08:30:06.889256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.889236    4898 generic.go:356] "Generic (PLEG): container finished" podID="d8398fcd-e412-4abe-b630-78573f4cf7fe" containerID="71c3a2f7c3f76a66e649731c183aa424b81adb5b42ca7d9a5e9938ceef233f5b" exitCode=60
Nov 02 08:30:06.889369 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.889316    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq" event={"ID":"d8398fcd-e412-4abe-b630-78573f4cf7fe","Type":"ContainerDied","Data":"71c3a2f7c3f76a66e649731c183aa424b81adb5b42ca7d9a5e9938ceef233f5b"}
Nov 02 08:30:06.889525 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.889511    4898 scope.go:122] "RemoveContainer" containerID="71c3a2f7c3f76a66e649731c183aa424b81adb5b42ca7d9a5e9938ceef233f5b"
Nov 02 08:30:06.889768 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:06.889748    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq_odh-ai-gateway-infra(d8398fcd-e412-4abe-b630-78573f4cf7fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq" podUID="d8398fcd-e412-4abe-b630-78573f4cf7fe"
Nov 02 08:30:06.891310 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.891290    4898 scope.go:122] "RemoveContainer" containerID="f268d3c646cb58801872f0f2f70252e3dc34cff04021e111fe3e8aa6f23f0f12"
Nov 02 08:30:06.891573 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:06.891547    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97_odh-ai-gateway-infra(f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc"
Nov 02 08:30:06.893055 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.893032    4898 generic.go:356] "Generic (PLEG): container finished" podID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" containerID="08f85ada8a057343e53589634b3520a45dc940e4d6fa5e0ead01ba2e67c9e018" exitCode=60
Nov 02 08:30:06.893148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.893112    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t" event={"ID":"0e09d271-5894-4f0d-9a38-dca2b8efb7bb","Type":"ContainerDied","Data":"08f85ada8a057343e53589634b3520a45dc940e4d6fa5e0ead01ba2e67c9e018"}
Nov 02 08:30:06.893388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.893370    4898 scope.go:122] "RemoveContainer" containerID="08f85ada8a057343e53589634b3520a45dc940e4d6fa5e0ead01ba2e67c9e018"
Nov 02 08:30:06.893618 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:06.893589    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t_odh-ai-gateway-infra(0e09d271-5894-4f0d-9a38-dca2b8efb7bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t" podUID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb"
Nov 02 08:30:06.895269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.895245    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc" event={"ID":"eeb4d901-47b4-4247-b099-db9bd7144ce2","Type":"ContainerStarted","Data":"00dd46a535c978b4e579c6e4a2be1a61e0d8dd2fb5ffaede111acf5de3bdbbb2"}
Nov 02 08:30:06.903955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.903931    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx" event={"ID":"b0539990-8490-41d3-8dba-77f7e8a6af14","Type":"ContainerStarted","Data":"24b7a956d80cd07d94a3324c25d51dab813e505add8a5b8b7d8e7ef5794a44ab"}
Nov 02 08:30:06.906967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.906939    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw" event={"ID":"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32","Type":"ContainerStarted","Data":"2c0e657d9bb057bb43230beed3a89dca848d6a7a5f1ee7680c5fee9ec6912994"}
Nov 02 08:30:06.909024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.908999    4898 scope.go:122] "RemoveContainer" containerID="2356b15d53e3971669bcdbcc95490114dc7cafa874a9c474ee160916a304755d"
Nov 02 08:30:06.909145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.909122    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx" event={"ID":"045dedc0-929e-4caf-8349-11463c81fc48","Type":"ContainerStarted","Data":"2da25a1a875b06f000516ba8fc9069a3034eaaf0620712193ffbfed58c9bbce4"}
Nov 02 08:30:06.911140 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.911116    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv" event={"ID":"eb7a2f08-f10d-4430-b0be-60892b129043","Type":"ContainerStarted","Data":"00969c9d50231ff091df0a5d39a43a77e4862938bd1938032a6d190603c0703e"}
Nov 02 08:30:06.913138 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.913116    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn" event={"ID":"d85b69ee-2f64-45f1-af4e-9f23b122f2c8","Type":"ContainerStarted","Data":"fce9204829fad637e220db2dbf510b1209dbef7d5ef4625bcefc41eb2036ca32"}
Nov 02 08:30:06.915323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.915298    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w" event={"ID":"e9d7ab63-f8af-475d-a6e8-67e7d5587603","Type":"ContainerStarted","Data":"6992a4244769a085d3b8d8af5778e32b9ee6ec83c2c5738e0bf991609b80b092"}
Nov 02 08:30:06.917637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.917612    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq" event={"ID":"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab","Type":"ContainerStarted","Data":"b244467dd9f03acb1efa9f43625f9a7463387901611eef1b400fae070a70f6db"}
Nov 02 08:30:06.919617 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.919593    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq" event={"ID":"48b72892-6169-4ffa-bc4a-e64ad46688ae","Type":"ContainerStarted","Data":"d292179741c62ea7fb23a70f335a13e2c9f07afb6f96a5deb43f819d325ede34"}
Nov 02 08:30:06.925228 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:06.925205    4898 scope.go:122] "RemoveContainer" containerID="c6096d223d009868d74103e756f670a342392cde76a4c0ce453cf8db665ab005"
Nov 02 08:30:07.828610 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.828571    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"]
Nov 02 08:30:07.902136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.902108    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:30:07.936307 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.936280    4898 scope.go:122] "RemoveContainer" containerID="4744a29e1cb5b93578cc071c8c3be91bcc1b6b712a48a4c81c46bf5fcc2a95e6"
Nov 02 08:30:07.936711 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.936680    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp_odh-ai-gateway-infra(a259b571-ddfe-400f-a22b-7f16768a82f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7"
Nov 02 08:30:07.938336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.938310    4898 scope.go:122] "RemoveContainer" containerID="71c3a2f7c3f76a66e649731c183aa424b81adb5b42ca7d9a5e9938ceef233f5b"
Nov 02 08:30:07.938679 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.938657    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq_odh-ai-gateway-infra(d8398fcd-e412-4abe-b630-78573f4cf7fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq" podUID="d8398fcd-e412-4abe-b630-78573f4cf7fe"
Nov 02 08:30:07.940421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.940398    4898 scope.go:122] "RemoveContainer" containerID="08f85ada8a057343e53589634b3520a45dc940e4d6fa5e0ead01ba2e67c9e018"
Nov 02 08:30:07.940707 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.940680    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t_odh-ai-gateway-infra(0e09d271-5894-4f0d-9a38-dca2b8efb7bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t" podUID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb"
Nov 02 08:30:07.942485 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.942462    4898 generic.go:356] "Generic (PLEG): container finished" podID="eeb4d901-47b4-4247-b099-db9bd7144ce2" containerID="00dd46a535c978b4e579c6e4a2be1a61e0d8dd2fb5ffaede111acf5de3bdbbb2" exitCode=60
Nov 02 08:30:07.942579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.942541    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc" event={"ID":"eeb4d901-47b4-4247-b099-db9bd7144ce2","Type":"ContainerDied","Data":"00dd46a535c978b4e579c6e4a2be1a61e0d8dd2fb5ffaede111acf5de3bdbbb2"}
Nov 02 08:30:07.942654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.942605    4898 scope.go:122] "RemoveContainer" containerID="77ce45b52ce89b980aacafcd05bf72972a9e9364a18515013af86f69726fda7d"
Nov 02 08:30:07.942799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.942781    4898 scope.go:122] "RemoveContainer" containerID="00dd46a535c978b4e579c6e4a2be1a61e0d8dd2fb5ffaede111acf5de3bdbbb2"
Nov 02 08:30:07.943170 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.943143    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc_odh-ai-gateway-infra(eeb4d901-47b4-4247-b099-db9bd7144ce2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2"
Nov 02 08:30:07.945831 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.945804    4898 generic.go:356] "Generic (PLEG): container finished" podID="b0539990-8490-41d3-8dba-77f7e8a6af14" containerID="24b7a956d80cd07d94a3324c25d51dab813e505add8a5b8b7d8e7ef5794a44ab" exitCode=60
Nov 02 08:30:07.945997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.945913    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx" event={"ID":"b0539990-8490-41d3-8dba-77f7e8a6af14","Type":"ContainerDied","Data":"24b7a956d80cd07d94a3324c25d51dab813e505add8a5b8b7d8e7ef5794a44ab"}
Nov 02 08:30:07.946211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.946196    4898 scope.go:122] "RemoveContainer" containerID="24b7a956d80cd07d94a3324c25d51dab813e505add8a5b8b7d8e7ef5794a44ab"
Nov 02 08:30:07.946503 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.946468    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx_odh-ai-gateway-infra(b0539990-8490-41d3-8dba-77f7e8a6af14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx" podUID="b0539990-8490-41d3-8dba-77f7e8a6af14"
Nov 02 08:30:07.954685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.948491    4898 generic.go:356] "Generic (PLEG): container finished" podID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" containerID="2c0e657d9bb057bb43230beed3a89dca848d6a7a5f1ee7680c5fee9ec6912994" exitCode=60
Nov 02 08:30:07.954685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.948610    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw" event={"ID":"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32","Type":"ContainerDied","Data":"2c0e657d9bb057bb43230beed3a89dca848d6a7a5f1ee7680c5fee9ec6912994"}
Nov 02 08:30:07.954685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.949009    4898 scope.go:122] "RemoveContainer" containerID="2c0e657d9bb057bb43230beed3a89dca848d6a7a5f1ee7680c5fee9ec6912994"
Nov 02 08:30:07.954685 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.949300    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw_odh-ai-gateway-infra(6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32"
Nov 02 08:30:07.954685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.953320    4898 generic.go:356] "Generic (PLEG): container finished" podID="045dedc0-929e-4caf-8349-11463c81fc48" containerID="2da25a1a875b06f000516ba8fc9069a3034eaaf0620712193ffbfed58c9bbce4" exitCode=60
Nov 02 08:30:07.954685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.953605    4898 scope.go:122] "RemoveContainer" containerID="2da25a1a875b06f000516ba8fc9069a3034eaaf0620712193ffbfed58c9bbce4"
Nov 02 08:30:07.954685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.953687    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx" event={"ID":"045dedc0-929e-4caf-8349-11463c81fc48","Type":"ContainerDied","Data":"2da25a1a875b06f000516ba8fc9069a3034eaaf0620712193ffbfed58c9bbce4"}
Nov 02 08:30:07.954685 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.953918    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx_odh-ai-gateway-infra(045dedc0-929e-4caf-8349-11463c81fc48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx" podUID="045dedc0-929e-4caf-8349-11463c81fc48"
Nov 02 08:30:07.956242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.956215    4898 generic.go:356] "Generic (PLEG): container finished" podID="eb7a2f08-f10d-4430-b0be-60892b129043" containerID="00969c9d50231ff091df0a5d39a43a77e4862938bd1938032a6d190603c0703e" exitCode=60
Nov 02 08:30:07.956362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.956329    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv" event={"ID":"eb7a2f08-f10d-4430-b0be-60892b129043","Type":"ContainerDied","Data":"00969c9d50231ff091df0a5d39a43a77e4862938bd1938032a6d190603c0703e"}
Nov 02 08:30:07.956547 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.956527    4898 scope.go:122] "RemoveContainer" containerID="00969c9d50231ff091df0a5d39a43a77e4862938bd1938032a6d190603c0703e"
Nov 02 08:30:07.956821 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.956794    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv_odh-ai-gateway-infra(eb7a2f08-f10d-4430-b0be-60892b129043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv" podUID="eb7a2f08-f10d-4430-b0be-60892b129043"
Nov 02 08:30:07.959030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.959008    4898 generic.go:356] "Generic (PLEG): container finished" podID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" containerID="fce9204829fad637e220db2dbf510b1209dbef7d5ef4625bcefc41eb2036ca32" exitCode=60
Nov 02 08:30:07.959900 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.959399    4898 scope.go:122] "RemoveContainer" containerID="fce9204829fad637e220db2dbf510b1209dbef7d5ef4625bcefc41eb2036ca32"
Nov 02 08:30:07.959900 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.959091    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn" event={"ID":"d85b69ee-2f64-45f1-af4e-9f23b122f2c8","Type":"ContainerDied","Data":"fce9204829fad637e220db2dbf510b1209dbef7d5ef4625bcefc41eb2036ca32"}
Nov 02 08:30:07.959900 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.959642    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn_odh-ai-gateway-infra(d85b69ee-2f64-45f1-af4e-9f23b122f2c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8"
Nov 02 08:30:07.961104 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.961082    4898 generic.go:356] "Generic (PLEG): container finished" podID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerID="f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b" exitCode=0
Nov 02 08:30:07.961183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.961111    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" event={"ID":"8d3b67b4-66e9-4f49-942d-61b4b632ac82","Type":"ContainerDied","Data":"f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b"}
Nov 02 08:30:07.961183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.961145    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"
Nov 02 08:30:07.961183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.961168    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4" event={"ID":"8d3b67b4-66e9-4f49-942d-61b4b632ac82","Type":"ContainerDied","Data":"9b3306756cee18e89e3dcf7fbec6f15aba2eec04ee40f094c4d152b3d2567f89"}
Nov 02 08:30:07.961978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.961874    4898 scope.go:122] "RemoveContainer" containerID="12f73efd8ab1b08688312f4e2c3b4920d2525ccd76a9986e9a8b4bd9d33c2eaa"
Nov 02 08:30:07.963452 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.963433    4898 generic.go:356] "Generic (PLEG): container finished" podID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" containerID="6992a4244769a085d3b8d8af5778e32b9ee6ec83c2c5738e0bf991609b80b092" exitCode=60
Nov 02 08:30:07.963540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.963511    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w" event={"ID":"e9d7ab63-f8af-475d-a6e8-67e7d5587603","Type":"ContainerDied","Data":"6992a4244769a085d3b8d8af5778e32b9ee6ec83c2c5738e0bf991609b80b092"}
Nov 02 08:30:07.963768 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.963747    4898 scope.go:122] "RemoveContainer" containerID="6992a4244769a085d3b8d8af5778e32b9ee6ec83c2c5738e0bf991609b80b092"
Nov 02 08:30:07.964095 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.964072    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w_odh-ai-gateway-infra(e9d7ab63-f8af-475d-a6e8-67e7d5587603)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w" podUID="e9d7ab63-f8af-475d-a6e8-67e7d5587603"
Nov 02 08:30:07.965714 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.965688    4898 generic.go:356] "Generic (PLEG): container finished" podID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" containerID="b244467dd9f03acb1efa9f43625f9a7463387901611eef1b400fae070a70f6db" exitCode=60
Nov 02 08:30:07.966032 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.966010    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq" event={"ID":"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab","Type":"ContainerDied","Data":"b244467dd9f03acb1efa9f43625f9a7463387901611eef1b400fae070a70f6db"}
Nov 02 08:30:07.966216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.966133    4898 scope.go:122] "RemoveContainer" containerID="b244467dd9f03acb1efa9f43625f9a7463387901611eef1b400fae070a70f6db"
Nov 02 08:30:07.966446 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.966423    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq_odh-ai-gateway-infra(3109b9e6-5dfe-4422-a630-b9c42c6bc1ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab"
Nov 02 08:30:07.968371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.968347    4898 generic.go:356] "Generic (PLEG): container finished" podID="48b72892-6169-4ffa-bc4a-e64ad46688ae" containerID="d292179741c62ea7fb23a70f335a13e2c9f07afb6f96a5deb43f819d325ede34" exitCode=60
Nov 02 08:30:07.968499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.968482    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq" event={"ID":"48b72892-6169-4ffa-bc4a-e64ad46688ae","Type":"ContainerDied","Data":"d292179741c62ea7fb23a70f335a13e2c9f07afb6f96a5deb43f819d325ede34"}
Nov 02 08:30:07.968934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.968801    4898 scope.go:122] "RemoveContainer" containerID="d292179741c62ea7fb23a70f335a13e2c9f07afb6f96a5deb43f819d325ede34"
Nov 02 08:30:07.969110 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.969086    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq_odh-ai-gateway-infra(48b72892-6169-4ffa-bc4a-e64ad46688ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae"
Nov 02 08:30:07.971453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.971424    4898 generic.go:356] "Generic (PLEG): container finished" podID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" containerID="2ead689a5e2d48b4786d122736a6f1060764f24fb71bedba2408d0f965bf9341" exitCode=60
Nov 02 08:30:07.971566 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.971501    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv" event={"ID":"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f","Type":"ContainerDied","Data":"2ead689a5e2d48b4786d122736a6f1060764f24fb71bedba2408d0f965bf9341"}
Nov 02 08:30:07.971794 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.971777    4898 scope.go:122] "RemoveContainer" containerID="2ead689a5e2d48b4786d122736a6f1060764f24fb71bedba2408d0f965bf9341"
Nov 02 08:30:07.972062 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.972027    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv_odh-ai-gateway-infra(85f9a6a2-8c24-4edb-a91c-b2d906ddac6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f"
Nov 02 08:30:07.974333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.974313    4898 generic.go:356] "Generic (PLEG): container finished" podID="2abea0a3-c4f1-40df-af27-a75342569b3b" containerID="04864cbe72c260a18b98aa83e2f852cb841722c5b34010b65425dc680f92989f" exitCode=60
Nov 02 08:30:07.974423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.974347    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74" event={"ID":"2abea0a3-c4f1-40df-af27-a75342569b3b","Type":"ContainerDied","Data":"04864cbe72c260a18b98aa83e2f852cb841722c5b34010b65425dc680f92989f"}
Nov 02 08:30:07.974639 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:07.974621    4898 scope.go:122] "RemoveContainer" containerID="04864cbe72c260a18b98aa83e2f852cb841722c5b34010b65425dc680f92989f"
Nov 02 08:30:07.974847 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:07.974824    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74_odh-ai-gateway-infra(2abea0a3-c4f1-40df-af27-a75342569b3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b"
Nov 02 08:30:08.075298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.071920    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-kube-api-access-zzcml\" (UniqueName: \"kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-kube-api-access-zzcml\") pod \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") "
Nov 02 08:30:08.075298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.071989    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-certs\") pod \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") "
Nov 02 08:30:08.075298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.072067    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-token\" (UniqueName: \"kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-token\") pod \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") "
Nov 02 08:30:08.075298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.072186    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-socket\") pod \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") "
Nov 02 08:30:08.075298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.072341    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-podinfo\") pod \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") "
Nov 02 08:30:08.075298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.072376    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-envoy\") pod \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") "
Nov 02 08:30:08.075298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.072494    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-credential-socket\") pod \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") "
Nov 02 08:30:08.075298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.072535    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-data\") pod \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") "
Nov 02 08:30:08.075298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.072616    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istiod-ca-cert\") pod \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\" (UID: \"8d3b67b4-66e9-4f49-942d-61b4b632ac82\") "
Nov 02 08:30:08.124808 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.096866    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-certs" pod "8d3b67b4-66e9-4f49-942d-61b4b632ac82" (UID: "8d3b67b4-66e9-4f49-942d-61b4b632ac82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:08.124946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.112623    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-socket" pod "8d3b67b4-66e9-4f49-942d-61b4b632ac82" (UID: "8d3b67b4-66e9-4f49-942d-61b4b632ac82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:08.125046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.113967    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istiod-ca-cert" pod "8d3b67b4-66e9-4f49-942d-61b4b632ac82" (UID: "8d3b67b4-66e9-4f49-942d-61b4b632ac82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:08.125133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.115118    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-credential-socket" pod "8d3b67b4-66e9-4f49-942d-61b4b632ac82" (UID: "8d3b67b4-66e9-4f49-942d-61b4b632ac82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:08.125206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.120610    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-podinfo" pod "8d3b67b4-66e9-4f49-942d-61b4b632ac82" (UID: "8d3b67b4-66e9-4f49-942d-61b4b632ac82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:30:08.125288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.123506    4898 scope.go:122] "RemoveContainer" containerID="517c3d339113203ea1efc52bd062c9c9e0dc70b5fc7cf284edf6aa89bb0370a4"
Nov 02 08:30:08.125607 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.125586    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-data" pod "8d3b67b4-66e9-4f49-942d-61b4b632ac82" (UID: "8d3b67b4-66e9-4f49-942d-61b4b632ac82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:08.141721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.141690    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-token" pod "8d3b67b4-66e9-4f49-942d-61b4b632ac82" (UID: "8d3b67b4-66e9-4f49-942d-61b4b632ac82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:08.141954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.141915    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-kube-api-access-zzcml" pod "8d3b67b4-66e9-4f49-942d-61b4b632ac82" (UID: "8d3b67b4-66e9-4f49-942d-61b4b632ac82"). InnerVolumeSpecName "kube-api-access-zzcml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:08.145259 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.145230    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-envoy" pod "8d3b67b4-66e9-4f49-942d-61b4b632ac82" (UID: "8d3b67b4-66e9-4f49-942d-61b4b632ac82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:08.202403 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.202363    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:08.331216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.326493    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:08.331216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.326528    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:08.331216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.326542    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:08.331216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.326559    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:08.331216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.326573    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:08.331216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.326587    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzcml\" (UniqueName: \"kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-kube-api-access-zzcml\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:08.331216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.326601    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3b67b4-66e9-4f49-942d-61b4b632ac82-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:08.331216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.326613    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3b67b4-66e9-4f49-942d-61b4b632ac82-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:08.344557 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.344480    4898 scope.go:122] "RemoveContainer" containerID="b6d6bc8e4fe80ae6ec3702882d888be8f03f8122755722de3859f76c3d2127b3"
Nov 02 08:30:08.442312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.442287    4898 scope.go:122] "RemoveContainer" containerID="9167a25cbbbfd1f6585da5150080b6c233fb46875cac0e507deb2164679afbe5"
Nov 02 08:30:08.457301 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.457275    4898 scope.go:122] "RemoveContainer" containerID="dcf1e2ae0398f0ee0b206faf73711168d46b6dbe27b3de8c8fea0c9f72216b7c"
Nov 02 08:30:08.473517 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.473489    4898 scope.go:122] "RemoveContainer" containerID="f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b"
Nov 02 08:30:08.484850 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.484822    4898 scope.go:122] "RemoveContainer" containerID="5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963"
Nov 02 08:30:08.497930 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.497906    4898 scope.go:122] "RemoveContainer" containerID="f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b"
Nov 02 08:30:08.498312 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:08.498283    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b\": container with ID starting with f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b not found: ID does not exist" containerID="f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b"
Nov 02 08:30:08.498420 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.498323    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b"} err="failed to get container status \"f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b\": rpc error: code = NotFound desc = could not find container \"f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b\": container with ID starting with f925b19422f344bbbe81d8db35d9918668b9fca4e6a7fd1752067f6955727b7b not found: ID does not exist"
Nov 02 08:30:08.498420 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.498347    4898 scope.go:122] "RemoveContainer" containerID="5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963"
Nov 02 08:30:08.498624 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:08.498599    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963\": container with ID starting with 5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963 not found: ID does not exist" containerID="5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963"
Nov 02 08:30:08.498717 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.498632    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963"} err="failed to get container status \"5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963\": rpc error: code = NotFound desc = could not find container \"5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963\": container with ID starting with 5d1a1f3152df138db9d0e5a182ceb0eac512b15569528539948ea37f2c0bb963 not found: ID does not exist"
Nov 02 08:30:08.498717 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.498653    4898 scope.go:122] "RemoveContainer" containerID="666fab951ecc4db38fe68fcaed690c39a64a2d1ed068cdb847d15428f1c82002"
Nov 02 08:30:08.517181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.517157    4898 scope.go:122] "RemoveContainer" containerID="11a412179060025e30e1818f6d9c6eb5fdd5ea5cc3b94e9102a84774fc03864e"
Nov 02 08:30:08.532818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.532797    4898 scope.go:122] "RemoveContainer" containerID="81e2fcd619836f780d0b1e3a198470d2b2e572e9e1a32d0fee6ec8e226ad89d3"
Nov 02 08:30:08.547512 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.547493    4898 scope.go:122] "RemoveContainer" containerID="2ad84540c2e1b9493b629b4daa4aa051796992ec8bf0d8213ca1b5d61287df87"
Nov 02 08:30:08.562645 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.562557    4898 scope.go:122] "RemoveContainer" containerID="d328dd5f1b39cf0de4bd17dbeaf4a441b92ad5acc4807c37ab42ae43f3761d36"
Nov 02 08:30:08.808932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.808884    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"]
Nov 02 08:30:08.817674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.817640    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2dde84-openshift-default-799f6d654f-fw5c4"]
Nov 02 08:30:08.854766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.854732    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"]
Nov 02 08:30:08.994909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.994867    4898 scope.go:122] "RemoveContainer" containerID="2c0e657d9bb057bb43230beed3a89dca848d6a7a5f1ee7680c5fee9ec6912994"
Nov 02 08:30:08.995477 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:08.995374    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw_odh-ai-gateway-infra(6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32"
Nov 02 08:30:08.997116 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.997095    4898 scope.go:122] "RemoveContainer" containerID="2da25a1a875b06f000516ba8fc9069a3034eaaf0620712193ffbfed58c9bbce4"
Nov 02 08:30:08.997481 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:08.997453    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx_odh-ai-gateway-infra(045dedc0-929e-4caf-8349-11463c81fc48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx" podUID="045dedc0-929e-4caf-8349-11463c81fc48"
Nov 02 08:30:08.999077 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:08.999048    4898 scope.go:122] "RemoveContainer" containerID="00969c9d50231ff091df0a5d39a43a77e4862938bd1938032a6d190603c0703e"
Nov 02 08:30:08.999445 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:08.999407    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv_odh-ai-gateway-infra(eb7a2f08-f10d-4430-b0be-60892b129043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv" podUID="eb7a2f08-f10d-4430-b0be-60892b129043"
Nov 02 08:30:09.001103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.001083    4898 scope.go:122] "RemoveContainer" containerID="fce9204829fad637e220db2dbf510b1209dbef7d5ef4625bcefc41eb2036ca32"
Nov 02 08:30:09.001413 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:09.001383    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn_odh-ai-gateway-infra(d85b69ee-2f64-45f1-af4e-9f23b122f2c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8"
Nov 02 08:30:09.004582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.004562    4898 scope.go:122] "RemoveContainer" containerID="6992a4244769a085d3b8d8af5778e32b9ee6ec83c2c5738e0bf991609b80b092"
Nov 02 08:30:09.004842 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:09.004823    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w_odh-ai-gateway-infra(e9d7ab63-f8af-475d-a6e8-67e7d5587603)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w" podUID="e9d7ab63-f8af-475d-a6e8-67e7d5587603"
Nov 02 08:30:09.006642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.006620    4898 scope.go:122] "RemoveContainer" containerID="b244467dd9f03acb1efa9f43625f9a7463387901611eef1b400fae070a70f6db"
Nov 02 08:30:09.006980 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:09.006951    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq_odh-ai-gateway-infra(3109b9e6-5dfe-4422-a630-b9c42c6bc1ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab"
Nov 02 08:30:09.008369 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.008351    4898 scope.go:122] "RemoveContainer" containerID="d292179741c62ea7fb23a70f335a13e2c9f07afb6f96a5deb43f819d325ede34"
Nov 02 08:30:09.008587 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:09.008567    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq_odh-ai-gateway-infra(48b72892-6169-4ffa-bc4a-e64ad46688ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae"
Nov 02 08:30:09.010369 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.010348    4898 scope.go:122] "RemoveContainer" containerID="2ead689a5e2d48b4786d122736a6f1060764f24fb71bedba2408d0f965bf9341"
Nov 02 08:30:09.010649 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:09.010619    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv_odh-ai-gateway-infra(85f9a6a2-8c24-4edb-a91c-b2d906ddac6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f"
Nov 02 08:30:09.011864 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.011843    4898 scope.go:122] "RemoveContainer" containerID="04864cbe72c260a18b98aa83e2f852cb841722c5b34010b65425dc680f92989f"
Nov 02 08:30:09.012128 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:09.012102    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74_odh-ai-gateway-infra(2abea0a3-c4f1-40df-af27-a75342569b3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b"
Nov 02 08:30:09.013649 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.013631    4898 scope.go:122] "RemoveContainer" containerID="00dd46a535c978b4e579c6e4a2be1a61e0d8dd2fb5ffaede111acf5de3bdbbb2"
Nov 02 08:30:09.013883 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:09.013861    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc_odh-ai-gateway-infra(eeb4d901-47b4-4247-b099-db9bd7144ce2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2"
Nov 02 08:30:09.015636 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.015617    4898 scope.go:122] "RemoveContainer" containerID="24b7a956d80cd07d94a3324c25d51dab813e505add8a5b8b7d8e7ef5794a44ab"
Nov 02 08:30:09.015861 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:09.015837    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx_odh-ai-gateway-infra(b0539990-8490-41d3-8dba-77f7e8a6af14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx" podUID="b0539990-8490-41d3-8dba-77f7e8a6af14"
Nov 02 08:30:09.202183 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.202154    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"
Nov 02 08:30:09.386311 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.386219    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8398fcd-e412-4abe-b630-78573f4cf7fe-kube-api-access-g2jqt\" (UniqueName: \"kubernetes.io/projected/d8398fcd-e412-4abe-b630-78573f4cf7fe-kube-api-access-g2jqt\") pod \"d8398fcd-e412-4abe-b630-78573f4cf7fe\" (UID: \"d8398fcd-e412-4abe-b630-78573f4cf7fe\") "
Nov 02 08:30:09.386531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.386360    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8398fcd-e412-4abe-b630-78573f4cf7fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8398fcd-e412-4abe-b630-78573f4cf7fe-maas-api-service-ca\") pod \"d8398fcd-e412-4abe-b630-78573f4cf7fe\" (UID: \"d8398fcd-e412-4abe-b630-78573f4cf7fe\") "
Nov 02 08:30:09.386791 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.386766    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8398fcd-e412-4abe-b630-78573f4cf7fe-maas-api-service-ca" pod "d8398fcd-e412-4abe-b630-78573f4cf7fe" (UID: "d8398fcd-e412-4abe-b630-78573f4cf7fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:09.387076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.387056    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8398fcd-e412-4abe-b630-78573f4cf7fe-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:09.388628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.388602    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8398fcd-e412-4abe-b630-78573f4cf7fe-kube-api-access-g2jqt" pod "d8398fcd-e412-4abe-b630-78573f4cf7fe" (UID: "d8398fcd-e412-4abe-b630-78573f4cf7fe"). InnerVolumeSpecName "kube-api-access-g2jqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:09.494359 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.494318    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2jqt\" (UniqueName: \"kubernetes.io/projected/d8398fcd-e412-4abe-b630-78573f4cf7fe-kube-api-access-g2jqt\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:09.526681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:09.526642    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" path="/var/lib/kubelet/pods/8d3b67b4-66e9-4f49-942d-61b4b632ac82/volumes"
Nov 02 08:30:10.027360 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:10.027275    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"
Nov 02 08:30:10.027849 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:10.027284    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq" event={"ID":"d8398fcd-e412-4abe-b630-78573f4cf7fe","Type":"ContainerDied","Data":"66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33"}
Nov 02 08:30:10.027849 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:10.027460    4898 scope.go:122] "RemoveContainer" containerID="71c3a2f7c3f76a66e649731c183aa424b81adb5b42ca7d9a5e9938ceef233f5b"
Nov 02 08:30:10.081499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:10.081471    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"]
Nov 02 08:30:10.086166 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:10.086139    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0f2e06-29893470-gxtvq"]
Nov 02 08:30:11.164427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:11.164384    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"]
Nov 02 08:30:11.165002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:11.164611    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerName="istio-proxy" containerID="cri-o://24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f" gracePeriod=30
Nov 02 08:30:11.470860 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:11.470768    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8398fcd-e412-4abe-b630-78573f4cf7fe" path="/var/lib/kubelet/pods/d8398fcd-e412-4abe-b630-78573f4cf7fe/volumes"
Nov 02 08:30:12.599472 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:12.599425    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice/crio-66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:13.761505 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:13.761455    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice/crio-66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:13.855800 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:13.855758    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt" podUID="5379dd80-6ad8-4b2a-a49c-e012dfc0fab6" containerName="payload-pre-processing" containerID="cri-o://e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d" gracePeriod=25
Nov 02 08:30:14.991687 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:14.991643    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:30:16.456836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.456808    4898 scope.go:122] "RemoveContainer" containerID="f268d3c646cb58801872f0f2f70252e3dc34cff04021e111fe3e8aa6f23f0f12"
Nov 02 08:30:16.461979 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.461957    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:30:16.610228 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610192    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-credential-socket\") pod \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") "
Nov 02 08:30:16.610444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610242    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-podinfo\") pod \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") "
Nov 02 08:30:16.610444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610314    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-kube-api-access-2w674\" (UniqueName: \"kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-kube-api-access-2w674\") pod \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") "
Nov 02 08:30:16.610444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610392    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-certs\") pod \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") "
Nov 02 08:30:16.610614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610449    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-envoy\") pod \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") "
Nov 02 08:30:16.610614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610509    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-socket\") pod \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") "
Nov 02 08:30:16.610614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610517    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-credential-socket" pod "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" (UID: "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:16.610784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610643    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-data\") pod \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") "
Nov 02 08:30:16.610784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610731    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-certs" pod "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" (UID: "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:16.610784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610748    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-token\" (UniqueName: \"kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-token\") pod \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") "
Nov 02 08:30:16.610957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610817    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istiod-ca-cert\") pod \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\" (UID: \"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d\") "
Nov 02 08:30:16.610957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610844    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-socket" pod "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" (UID: "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:16.610957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.610931    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-data" pod "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" (UID: "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:16.611264 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.611243    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istiod-ca-cert" pod "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" (UID: "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:16.611701 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.611678    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:16.611828 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.611809    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:16.611948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.611832    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:16.611948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.611859    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:16.611948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.611881    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:16.612883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.612857    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-podinfo" pod "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" (UID: "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:30:16.613125 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.613099    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-envoy" pod "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" (UID: "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:16.613354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.613326    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-kube-api-access-2w674" pod "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" (UID: "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d"). InnerVolumeSpecName "kube-api-access-2w674". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:16.613463 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.613389    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-token" pod "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" (UID: "e70ee1e4-32ac-433e-80e5-4ab92fd76d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:16.713254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.713176    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:16.713254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.713209    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w674\" (UniqueName: \"kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-kube-api-access-2w674\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:16.713254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.713220    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:16.713254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:16.713229    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:17.117772 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.117681    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97" event={"ID":"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc","Type":"ContainerStarted","Data":"a9aa9c05fb88dd4cee54e941bada9a200ab65e17fa4663e869d54bdcb6318c44"}
Nov 02 08:30:17.119841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.119813    4898 generic.go:356] "Generic (PLEG): container finished" podID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerID="24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f" exitCode=0
Nov 02 08:30:17.120026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.119870    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" event={"ID":"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d","Type":"ContainerDied","Data":"24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f"}
Nov 02 08:30:17.120026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.119875    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"
Nov 02 08:30:17.120026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.119930    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms" event={"ID":"e70ee1e4-32ac-433e-80e5-4ab92fd76d0d","Type":"ContainerDied","Data":"5265d338182e784caadfa77c22c734c2a3bde491e42bd6ea7f61de7cda9b3cf3"}
Nov 02 08:30:17.120026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.119962    4898 scope.go:122] "RemoveContainer" containerID="24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f"
Nov 02 08:30:17.132708 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.132687    4898 scope.go:122] "RemoveContainer" containerID="7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d"
Nov 02 08:30:17.143418 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.143395    4898 scope.go:122] "RemoveContainer" containerID="24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f"
Nov 02 08:30:17.143679 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:17.143655    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f\": container with ID starting with 24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f not found: ID does not exist" containerID="24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f"
Nov 02 08:30:17.143734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.143687    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f"} err="failed to get container status \"24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f\": rpc error: code = NotFound desc = could not find container \"24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f\": container with ID starting with 24c39b5150bd63276bd2b12466ff59fa6d84aeb9d2d55c98d83cfb2601c1889f not found: ID does not exist"
Nov 02 08:30:17.143734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.143704    4898 scope.go:122] "RemoveContainer" containerID="7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d"
Nov 02 08:30:17.143955 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:17.143936    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d\": container with ID starting with 7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d not found: ID does not exist" containerID="7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d"
Nov 02 08:30:17.143998 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.143960    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d"} err="failed to get container status \"7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d\": rpc error: code = NotFound desc = could not find container \"7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d\": container with ID starting with 7857f28ae188289765badf7f69622692fdae5a94420cd5112b06b9d61efa1b3d not found: ID does not exist"
Nov 02 08:30:17.201033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.201000    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"]
Nov 02 08:30:17.212469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.212439    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f2e06-openshift-default-8d5cccd8f-2j7ms"]
Nov 02 08:30:17.469257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:17.469112    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" path="/var/lib/kubelet/pods/e70ee1e4-32ac-433e-80e5-4ab92fd76d0d/volumes"
Nov 02 08:30:18.131192 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:18.131156    4898 generic.go:356] "Generic (PLEG): container finished" podID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" containerID="a9aa9c05fb88dd4cee54e941bada9a200ab65e17fa4663e869d54bdcb6318c44" exitCode=60
Nov 02 08:30:18.131367 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:18.131225    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97" event={"ID":"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc","Type":"ContainerDied","Data":"a9aa9c05fb88dd4cee54e941bada9a200ab65e17fa4663e869d54bdcb6318c44"}
Nov 02 08:30:18.131367 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:18.131285    4898 scope.go:122] "RemoveContainer" containerID="f268d3c646cb58801872f0f2f70252e3dc34cff04021e111fe3e8aa6f23f0f12"
Nov 02 08:30:18.131706 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:18.131690    4898 scope.go:122] "RemoveContainer" containerID="a9aa9c05fb88dd4cee54e941bada9a200ab65e17fa4663e869d54bdcb6318c44"
Nov 02 08:30:18.132112 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:18.132086    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97_odh-ai-gateway-infra(f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc"
Nov 02 08:30:18.182955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:18.182913    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"]
Nov 02 08:30:18.457272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:18.457239    4898 scope.go:122] "RemoveContainer" containerID="04864cbe72c260a18b98aa83e2f852cb841722c5b34010b65425dc680f92989f"
Nov 02 08:30:18.457537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:18.457511    4898 scope.go:122] "RemoveContainer" containerID="b244467dd9f03acb1efa9f43625f9a7463387901611eef1b400fae070a70f6db"
Nov 02 08:30:18.457813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:18.457792    4898 scope.go:122] "RemoveContainer" containerID="2c0e657d9bb057bb43230beed3a89dca848d6a7a5f1ee7680c5fee9ec6912994"
Nov 02 08:30:18.458026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:18.458007    4898 scope.go:122] "RemoveContainer" containerID="24b7a956d80cd07d94a3324c25d51dab813e505add8a5b8b7d8e7ef5794a44ab"
Nov 02 08:30:18.458400 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:18.458369    4898 scope.go:122] "RemoveContainer" containerID="2ead689a5e2d48b4786d122736a6f1060764f24fb71bedba2408d0f965bf9341"
Nov 02 08:30:19.141601 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.141561    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx" event={"ID":"b0539990-8490-41d3-8dba-77f7e8a6af14","Type":"ContainerStarted","Data":"c0f2bc72e1bb8f37384c9eebd306862d3fab5dfafdfd26d58ebf21d4fb5618d7"}
Nov 02 08:30:19.143899 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.143855    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw" event={"ID":"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32","Type":"ContainerStarted","Data":"402d5acc84634473ba7cb58f579eb77a68365bbb9c8eb7014d1b6af3be4cad61"}
Nov 02 08:30:19.146153 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.146127    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq" event={"ID":"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab","Type":"ContainerStarted","Data":"00bb186e1fdaa2869194126414e6d215a526691a479aadc564cd165f3f5a2da0"}
Nov 02 08:30:19.148217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.148182    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv" event={"ID":"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f","Type":"ContainerStarted","Data":"864acbb02babcd638a05b92e6c28473e95d3e85f71dc24b0840919fca976f389"}
Nov 02 08:30:19.150615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.150591    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74" event={"ID":"2abea0a3-c4f1-40df-af27-a75342569b3b","Type":"ContainerStarted","Data":"8d23bc74e49a32153d6a86eda678d6eea96ed46c15eb9156c00695df75c2ff61"}
Nov 02 08:30:19.401666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.399079    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"
Nov 02 08:30:19.466455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.466419    4898 scope.go:122] "RemoveContainer" containerID="4744a29e1cb5b93578cc071c8c3be91bcc1b6b712a48a4c81c46bf5fcc2a95e6"
Nov 02 08:30:19.466787 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.466760    4898 scope.go:122] "RemoveContainer" containerID="fce9204829fad637e220db2dbf510b1209dbef7d5ef4625bcefc41eb2036ca32"
Nov 02 08:30:19.466954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.466930    4898 scope.go:122] "RemoveContainer" containerID="2da25a1a875b06f000516ba8fc9069a3034eaaf0620712193ffbfed58c9bbce4"
Nov 02 08:30:19.467102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.467080    4898 scope.go:122] "RemoveContainer" containerID="6992a4244769a085d3b8d8af5778e32b9ee6ec83c2c5738e0bf991609b80b092"
Nov 02 08:30:19.518913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.516503    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"]
Nov 02 08:30:19.547591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.547565    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"]
Nov 02 08:30:19.554911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.554852    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-maas-api-service-ca\") pod \"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc\" (UID: \"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc\") "
Nov 02 08:30:19.555041 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.555018    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-kube-api-access-s9zqh\" (UniqueName: \"kubernetes.io/projected/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-kube-api-access-s9zqh\") pod \"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc\" (UID: \"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc\") "
Nov 02 08:30:19.555358 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.555320    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-maas-api-service-ca" pod "f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" (UID: "f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:19.559223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.556502    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:19.561573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.559440    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-kube-api-access-s9zqh" pod "f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" (UID: "f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc"). InnerVolumeSpecName "kube-api-access-s9zqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:19.663083 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:19.663027    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9zqh\" (UniqueName: \"kubernetes.io/projected/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc-kube-api-access-s9zqh\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:20.214968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.214219    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn" event={"ID":"d85b69ee-2f64-45f1-af4e-9f23b122f2c8","Type":"ContainerStarted","Data":"699c8ae5634bec7aec20ecced6603cfe3df666f9cb956641ff63be6305d97e58"}
Nov 02 08:30:20.231915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.228210    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w" event={"ID":"e9d7ab63-f8af-475d-a6e8-67e7d5587603","Type":"ContainerStarted","Data":"cb74ef647776a45edf5d15914698ec91ed70c1b1dfbf2522cd199b166763f072"}
Nov 02 08:30:20.242047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.238161    4898 generic.go:356] "Generic (PLEG): container finished" podID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" containerID="00bb186e1fdaa2869194126414e6d215a526691a479aadc564cd165f3f5a2da0" exitCode=60
Nov 02 08:30:20.242047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.238367    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq" event={"ID":"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab","Type":"ContainerDied","Data":"00bb186e1fdaa2869194126414e6d215a526691a479aadc564cd165f3f5a2da0"}
Nov 02 08:30:20.242047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.238426    4898 scope.go:122] "RemoveContainer" containerID="b244467dd9f03acb1efa9f43625f9a7463387901611eef1b400fae070a70f6db"
Nov 02 08:30:20.243099 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.242978    4898 scope.go:122] "RemoveContainer" containerID="00bb186e1fdaa2869194126414e6d215a526691a479aadc564cd165f3f5a2da0"
Nov 02 08:30:20.256985 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:20.250708    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq_odh-ai-gateway-infra(3109b9e6-5dfe-4422-a630-b9c42c6bc1ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab"
Nov 02 08:30:20.259505 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.258288    4898 generic.go:356] "Generic (PLEG): container finished" podID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" containerID="864acbb02babcd638a05b92e6c28473e95d3e85f71dc24b0840919fca976f389" exitCode=60
Nov 02 08:30:20.259505 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.258391    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv" event={"ID":"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f","Type":"ContainerDied","Data":"864acbb02babcd638a05b92e6c28473e95d3e85f71dc24b0840919fca976f389"}
Nov 02 08:30:20.259505 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.258910    4898 scope.go:122] "RemoveContainer" containerID="864acbb02babcd638a05b92e6c28473e95d3e85f71dc24b0840919fca976f389"
Nov 02 08:30:20.259505 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:20.259359    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv_odh-ai-gateway-infra(85f9a6a2-8c24-4edb-a91c-b2d906ddac6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f"
Nov 02 08:30:20.265411 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.265386    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"]
Nov 02 08:30:20.283091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.281359    4898 generic.go:356] "Generic (PLEG): container finished" podID="2abea0a3-c4f1-40df-af27-a75342569b3b" containerID="8d23bc74e49a32153d6a86eda678d6eea96ed46c15eb9156c00695df75c2ff61" exitCode=60
Nov 02 08:30:20.283091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.281504    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74" event={"ID":"2abea0a3-c4f1-40df-af27-a75342569b3b","Type":"ContainerDied","Data":"8d23bc74e49a32153d6a86eda678d6eea96ed46c15eb9156c00695df75c2ff61"}
Nov 02 08:30:20.283091 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.282097    4898 scope.go:122] "RemoveContainer" containerID="8d23bc74e49a32153d6a86eda678d6eea96ed46c15eb9156c00695df75c2ff61"
Nov 02 08:30:20.283091 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:20.282528    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74_odh-ai-gateway-infra(2abea0a3-c4f1-40df-af27-a75342569b3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b"
Nov 02 08:30:20.329922 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.320021    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"]
Nov 02 08:30:20.329922 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.323020    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp" event={"ID":"a259b571-ddfe-400f-a22b-7f16768a82f7","Type":"ContainerStarted","Data":"cee73badebe2fbcf0e45fa4a577af0fa024f0405c8a4c138a80437e8eed73252"}
Nov 02 08:30:20.339926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.337490    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97" event={"ID":"f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc","Type":"ContainerDied","Data":"7a4037d23994126d9c9ae5ee43adcd378e59dc19d0a47679279d0468def99f66"}
Nov 02 08:30:20.339926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.337604    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"
Nov 02 08:30:20.351777 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.351738    4898 scope.go:122] "RemoveContainer" containerID="2ead689a5e2d48b4786d122736a6f1060764f24fb71bedba2408d0f965bf9341"
Nov 02 08:30:20.368684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.367844    4898 generic.go:356] "Generic (PLEG): container finished" podID="b0539990-8490-41d3-8dba-77f7e8a6af14" containerID="c0f2bc72e1bb8f37384c9eebd306862d3fab5dfafdfd26d58ebf21d4fb5618d7" exitCode=60
Nov 02 08:30:20.368684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.368018    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx" event={"ID":"b0539990-8490-41d3-8dba-77f7e8a6af14","Type":"ContainerDied","Data":"c0f2bc72e1bb8f37384c9eebd306862d3fab5dfafdfd26d58ebf21d4fb5618d7"}
Nov 02 08:30:20.370578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.370450    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"]
Nov 02 08:30:20.380368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.378419    4898 generic.go:356] "Generic (PLEG): container finished" podID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" containerID="402d5acc84634473ba7cb58f579eb77a68365bbb9c8eb7014d1b6af3be4cad61" exitCode=60
Nov 02 08:30:20.380368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.378457    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw" event={"ID":"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32","Type":"ContainerDied","Data":"402d5acc84634473ba7cb58f579eb77a68365bbb9c8eb7014d1b6af3be4cad61"}
Nov 02 08:30:20.380368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.378821    4898 scope.go:122] "RemoveContainer" containerID="402d5acc84634473ba7cb58f579eb77a68365bbb9c8eb7014d1b6af3be4cad61"
Nov 02 08:30:20.380368 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:20.380319    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw_odh-ai-gateway-infra(6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32"
Nov 02 08:30:20.389916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.389259    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"]
Nov 02 08:30:20.394481 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.393056    4898 scope.go:122] "RemoveContainer" containerID="04864cbe72c260a18b98aa83e2f852cb841722c5b34010b65425dc680f92989f"
Nov 02 08:30:20.408354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.408027    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"]
Nov 02 08:30:20.419485 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.419458    4898 scope.go:122] "RemoveContainer" containerID="a9aa9c05fb88dd4cee54e941bada9a200ab65e17fa4663e869d54bdcb6318c44"
Nov 02 08:30:20.432284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.432198    4898 scope.go:122] "RemoveContainer" containerID="24b7a956d80cd07d94a3324c25d51dab813e505add8a5b8b7d8e7ef5794a44ab"
Nov 02 08:30:20.450923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.450861    4898 scope.go:122] "RemoveContainer" containerID="2c0e657d9bb057bb43230beed3a89dca848d6a7a5f1ee7680c5fee9ec6912994"
Nov 02 08:30:20.459647 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.459619    4898 scope.go:122] "RemoveContainer" containerID="00969c9d50231ff091df0a5d39a43a77e4862938bd1938032a6d190603c0703e"
Nov 02 08:30:20.627366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.627300    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp" podStartSLOduration=20.627279028 podStartE2EDuration="20.627279028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:30:20.620188695 +0000 UTC m=+3971.310515686" watchObservedRunningTime="2026-11-02 08:30:20.627279028 +0000 UTC m=+3971.317605993"
Nov 02 08:30:20.640865 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.640833    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"
Nov 02 08:30:20.665997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.665960    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"]
Nov 02 08:30:20.670454 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.670422    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c712b1-29893470-7td97"]
Nov 02 08:30:20.737199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.737106    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0539990-8490-41d3-8dba-77f7e8a6af14-kube-api-access-ggc7c\" (UniqueName: \"kubernetes.io/projected/b0539990-8490-41d3-8dba-77f7e8a6af14-kube-api-access-ggc7c\") pod \"b0539990-8490-41d3-8dba-77f7e8a6af14\" (UID: \"b0539990-8490-41d3-8dba-77f7e8a6af14\") "
Nov 02 08:30:20.737453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.737428    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0539990-8490-41d3-8dba-77f7e8a6af14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0539990-8490-41d3-8dba-77f7e8a6af14-maas-api-service-ca\") pod \"b0539990-8490-41d3-8dba-77f7e8a6af14\" (UID: \"b0539990-8490-41d3-8dba-77f7e8a6af14\") "
Nov 02 08:30:20.737858 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.737822    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0539990-8490-41d3-8dba-77f7e8a6af14-maas-api-service-ca" pod "b0539990-8490-41d3-8dba-77f7e8a6af14" (UID: "b0539990-8490-41d3-8dba-77f7e8a6af14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:20.738085 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.738060    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0539990-8490-41d3-8dba-77f7e8a6af14-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:20.754385 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.754340    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt" podUID="5379dd80-6ad8-4b2a-a49c-e012dfc0fab6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:30:20.771473 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.771427    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0539990-8490-41d3-8dba-77f7e8a6af14-kube-api-access-ggc7c" pod "b0539990-8490-41d3-8dba-77f7e8a6af14" (UID: "b0539990-8490-41d3-8dba-77f7e8a6af14"). InnerVolumeSpecName "kube-api-access-ggc7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:20.840539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:20.840494    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggc7c\" (UniqueName: \"kubernetes.io/projected/b0539990-8490-41d3-8dba-77f7e8a6af14-kube-api-access-ggc7c\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:21.312171 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.312072    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"]
Nov 02 08:30:21.392582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.392549    4898 generic.go:356] "Generic (PLEG): container finished" podID="045dedc0-929e-4caf-8349-11463c81fc48" containerID="22758c59ad1b0a7cc7a27eb77fe25ae020119ed2c43de112698d5b0095e1ab4a" exitCode=60
Nov 02 08:30:21.392782 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.392641    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx" event={"ID":"045dedc0-929e-4caf-8349-11463c81fc48","Type":"ContainerDied","Data":"22758c59ad1b0a7cc7a27eb77fe25ae020119ed2c43de112698d5b0095e1ab4a"}
Nov 02 08:30:21.392782 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.392717    4898 scope.go:122] "RemoveContainer" containerID="2da25a1a875b06f000516ba8fc9069a3034eaaf0620712193ffbfed58c9bbce4"
Nov 02 08:30:21.393147 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.393130    4898 scope.go:122] "RemoveContainer" containerID="22758c59ad1b0a7cc7a27eb77fe25ae020119ed2c43de112698d5b0095e1ab4a"
Nov 02 08:30:21.393492 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:21.393471    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx_odh-ai-gateway-infra(045dedc0-929e-4caf-8349-11463c81fc48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx" podUID="045dedc0-929e-4caf-8349-11463c81fc48"
Nov 02 08:30:21.396073 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.396045    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv" event={"ID":"eb7a2f08-f10d-4430-b0be-60892b129043","Type":"ContainerStarted","Data":"f99cc76d79065b1a679405b7a03740cba6534dc32a2bcbd91c23673b5b511d64"}
Nov 02 08:30:21.398522 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.398493    4898 generic.go:356] "Generic (PLEG): container finished" podID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" containerID="699c8ae5634bec7aec20ecced6603cfe3df666f9cb956641ff63be6305d97e58" exitCode=60
Nov 02 08:30:21.398657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.398571    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn" event={"ID":"d85b69ee-2f64-45f1-af4e-9f23b122f2c8","Type":"ContainerDied","Data":"699c8ae5634bec7aec20ecced6603cfe3df666f9cb956641ff63be6305d97e58"}
Nov 02 08:30:21.399017 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.398994    4898 scope.go:122] "RemoveContainer" containerID="699c8ae5634bec7aec20ecced6603cfe3df666f9cb956641ff63be6305d97e58"
Nov 02 08:30:21.399379 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:21.399354    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn_odh-ai-gateway-infra(d85b69ee-2f64-45f1-af4e-9f23b122f2c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8"
Nov 02 08:30:21.401853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.401833    4898 generic.go:356] "Generic (PLEG): container finished" podID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" containerID="cb74ef647776a45edf5d15914698ec91ed70c1b1dfbf2522cd199b166763f072" exitCode=60
Nov 02 08:30:21.402013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.401921    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w" event={"ID":"e9d7ab63-f8af-475d-a6e8-67e7d5587603","Type":"ContainerDied","Data":"cb74ef647776a45edf5d15914698ec91ed70c1b1dfbf2522cd199b166763f072"}
Nov 02 08:30:21.416444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.416345    4898 scope.go:122] "RemoveContainer" containerID="fce9204829fad637e220db2dbf510b1209dbef7d5ef4625bcefc41eb2036ca32"
Nov 02 08:30:21.428556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.428520    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"]
Nov 02 08:30:21.430217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.430036    4898 generic.go:356] "Generic (PLEG): container finished" podID="a259b571-ddfe-400f-a22b-7f16768a82f7" containerID="cee73badebe2fbcf0e45fa4a577af0fa024f0405c8a4c138a80437e8eed73252" exitCode=60
Nov 02 08:30:21.430217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.430110    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp" event={"ID":"a259b571-ddfe-400f-a22b-7f16768a82f7","Type":"ContainerDied","Data":"cee73badebe2fbcf0e45fa4a577af0fa024f0405c8a4c138a80437e8eed73252"}
Nov 02 08:30:21.431159 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.430598    4898 scope.go:122] "RemoveContainer" containerID="cee73badebe2fbcf0e45fa4a577af0fa024f0405c8a4c138a80437e8eed73252"
Nov 02 08:30:21.431159 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:21.431057    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp_odh-ai-gateway-infra(a259b571-ddfe-400f-a22b-7f16768a82f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7"
Nov 02 08:30:21.456347 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.454367    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx" event={"ID":"b0539990-8490-41d3-8dba-77f7e8a6af14","Type":"ContainerDied","Data":"0258898f681d6169d7cf6a015dd58fd0e3c158d50b2c07f8c7e686a13069c661"}
Nov 02 08:30:21.456347 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.454481    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"
Nov 02 08:30:21.466168 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.466142    4898 scope.go:122] "RemoveContainer" containerID="6992a4244769a085d3b8d8af5778e32b9ee6ec83c2c5738e0bf991609b80b092"
Nov 02 08:30:21.478302 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.478276    4898 scope.go:122] "RemoveContainer" containerID="d292179741c62ea7fb23a70f335a13e2c9f07afb6f96a5deb43f819d325ede34"
Nov 02 08:30:21.479544 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.479464    4898 scope.go:122] "RemoveContainer" containerID="08f85ada8a057343e53589634b3520a45dc940e4d6fa5e0ead01ba2e67c9e018"
Nov 02 08:30:21.482221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.482203    4898 scope.go:122] "RemoveContainer" containerID="00dd46a535c978b4e579c6e4a2be1a61e0d8dd2fb5ffaede111acf5de3bdbbb2"
Nov 02 08:30:21.653679 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.653643    4898 scope.go:122] "RemoveContainer" containerID="4744a29e1cb5b93578cc071c8c3be91bcc1b6b712a48a4c81c46bf5fcc2a95e6"
Nov 02 08:30:21.691913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.683174    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" path="/var/lib/kubelet/pods/f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc/volumes"
Nov 02 08:30:21.707352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.707322    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"]
Nov 02 08:30:21.894489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.886038    4898 scope.go:122] "RemoveContainer" containerID="c0f2bc72e1bb8f37384c9eebd306862d3fab5dfafdfd26d58ebf21d4fb5618d7"
Nov 02 08:30:21.943713 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.943682    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"
Nov 02 08:30:21.945958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.945928    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"
Nov 02 08:30:21.951684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.951656    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"
Nov 02 08:30:21.960421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.960393    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"
Nov 02 08:30:21.967863 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.967809    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"
Nov 02 08:30:21.982508 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.982472    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"]
Nov 02 08:30:21.989560 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:21.989532    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2365fa-29893470-qptkx"]
Nov 02 08:30:22.044678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.044640    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d7ab63-f8af-475d-a6e8-67e7d5587603-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d7ab63-f8af-475d-a6e8-67e7d5587603-maas-api-service-ca\") pod \"e9d7ab63-f8af-475d-a6e8-67e7d5587603\" (UID: \"e9d7ab63-f8af-475d-a6e8-67e7d5587603\") "
Nov 02 08:30:22.044938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.044799    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2abea0a3-c4f1-40df-af27-a75342569b3b-kube-api-access-vtzt9\" (UniqueName: \"kubernetes.io/projected/2abea0a3-c4f1-40df-af27-a75342569b3b-kube-api-access-vtzt9\") pod \"2abea0a3-c4f1-40df-af27-a75342569b3b\" (UID: \"2abea0a3-c4f1-40df-af27-a75342569b3b\") "
Nov 02 08:30:22.045040 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.044944    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2abea0a3-c4f1-40df-af27-a75342569b3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2abea0a3-c4f1-40df-af27-a75342569b3b-maas-api-service-ca\") pod \"2abea0a3-c4f1-40df-af27-a75342569b3b\" (UID: \"2abea0a3-c4f1-40df-af27-a75342569b3b\") "
Nov 02 08:30:22.045040 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.044994    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-kube-api-access-sfnc6\" (UniqueName: \"kubernetes.io/projected/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-kube-api-access-sfnc6\") pod \"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f\" (UID: \"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f\") "
Nov 02 08:30:22.045175 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.045048    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-maas-api-service-ca\") pod \"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab\" (UID: \"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab\") "
Nov 02 08:30:22.045175 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.045066    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d7ab63-f8af-475d-a6e8-67e7d5587603-maas-api-service-ca" pod "e9d7ab63-f8af-475d-a6e8-67e7d5587603" (UID: "e9d7ab63-f8af-475d-a6e8-67e7d5587603"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:22.045175 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.045166    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-maas-api-service-ca\") pod \"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f\" (UID: \"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f\") "
Nov 02 08:30:22.045347 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.045232    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-maas-api-service-ca\") pod \"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32\" (UID: \"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32\") "
Nov 02 08:30:22.045347 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.045266    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-kube-api-access-xrzwn\" (UniqueName: \"kubernetes.io/projected/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-kube-api-access-xrzwn\") pod \"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32\" (UID: \"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32\") "
Nov 02 08:30:22.045347 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.045299    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-kube-api-access-4jx4v\" (UniqueName: \"kubernetes.io/projected/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-kube-api-access-4jx4v\") pod \"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab\" (UID: \"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab\") "
Nov 02 08:30:22.045347 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.045325    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d7ab63-f8af-475d-a6e8-67e7d5587603-kube-api-access-q8zn6\" (UniqueName: \"kubernetes.io/projected/e9d7ab63-f8af-475d-a6e8-67e7d5587603-kube-api-access-q8zn6\") pod \"e9d7ab63-f8af-475d-a6e8-67e7d5587603\" (UID: \"e9d7ab63-f8af-475d-a6e8-67e7d5587603\") "
Nov 02 08:30:22.045625 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.045414    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abea0a3-c4f1-40df-af27-a75342569b3b-maas-api-service-ca" pod "2abea0a3-c4f1-40df-af27-a75342569b3b" (UID: "2abea0a3-c4f1-40df-af27-a75342569b3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:22.045873 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.045846    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-maas-api-service-ca" pod "6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" (UID: "6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:22.046263 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.046235    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-maas-api-service-ca" pod "3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" (UID: "3109b9e6-5dfe-4422-a630-b9c42c6bc1ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:22.046603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.046578    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-maas-api-service-ca" pod "85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" (UID: "85f9a6a2-8c24-4edb-a91c-b2d906ddac6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:22.047265 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.047231    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d7ab63-f8af-475d-a6e8-67e7d5587603-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.047382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.047269    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2abea0a3-c4f1-40df-af27-a75342569b3b-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.047382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.047285    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.047382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.047300    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.047382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.047315    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.069723 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.059150    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-kube-api-access-sfnc6" pod "85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" (UID: "85f9a6a2-8c24-4edb-a91c-b2d906ddac6f"). InnerVolumeSpecName "kube-api-access-sfnc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:22.069723 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.059334    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abea0a3-c4f1-40df-af27-a75342569b3b-kube-api-access-vtzt9" pod "2abea0a3-c4f1-40df-af27-a75342569b3b" (UID: "2abea0a3-c4f1-40df-af27-a75342569b3b"). InnerVolumeSpecName "kube-api-access-vtzt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:22.069723 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.063479    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-kube-api-access-xrzwn" pod "6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" (UID: "6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32"). InnerVolumeSpecName "kube-api-access-xrzwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:22.069723 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.063999    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d7ab63-f8af-475d-a6e8-67e7d5587603-kube-api-access-q8zn6" pod "e9d7ab63-f8af-475d-a6e8-67e7d5587603" (UID: "e9d7ab63-f8af-475d-a6e8-67e7d5587603"). InnerVolumeSpecName "kube-api-access-q8zn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:22.084358 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.084315    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-kube-api-access-4jx4v" pod "3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" (UID: "3109b9e6-5dfe-4422-a630-b9c42c6bc1ab"). InnerVolumeSpecName "kube-api-access-4jx4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:22.149354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.149266    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfnc6\" (UniqueName: \"kubernetes.io/projected/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f-kube-api-access-sfnc6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.149354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.149310    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrzwn\" (UniqueName: \"kubernetes.io/projected/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32-kube-api-access-xrzwn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.149354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.149329    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jx4v\" (UniqueName: \"kubernetes.io/projected/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab-kube-api-access-4jx4v\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.149354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.149343    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8zn6\" (UniqueName: \"kubernetes.io/projected/e9d7ab63-f8af-475d-a6e8-67e7d5587603-kube-api-access-q8zn6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.149625 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.149358    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtzt9\" (UniqueName: \"kubernetes.io/projected/2abea0a3-c4f1-40df-af27-a75342569b3b-kube-api-access-vtzt9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.468638 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.468598    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t" event={"ID":"0e09d271-5894-4f0d-9a38-dca2b8efb7bb","Type":"ContainerStarted","Data":"9d3226124d6627098db5ca2ebf74fd5d606ed1fcae707d81aa4e08cebe1c225c"}
Nov 02 08:30:22.471447 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.471118    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc" event={"ID":"eeb4d901-47b4-4247-b099-db9bd7144ce2","Type":"ContainerStarted","Data":"ecbef460d498f286f7541a68d4b113775a3d7dd558329162afd8d8eb48625d97"}
Nov 02 08:30:22.474654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.474630    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"
Nov 02 08:30:22.474654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.474639    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw" event={"ID":"6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32","Type":"ContainerDied","Data":"17f7cd13d1ed60f766a6959927dce2adab4269555a221d1ae2202d955c92a290"}
Nov 02 08:30:22.474843 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.474699    4898 scope.go:122] "RemoveContainer" containerID="402d5acc84634473ba7cb58f579eb77a68365bbb9c8eb7014d1b6af3be4cad61"
Nov 02 08:30:22.480366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.480190    4898 generic.go:356] "Generic (PLEG): container finished" podID="eb7a2f08-f10d-4430-b0be-60892b129043" containerID="f99cc76d79065b1a679405b7a03740cba6534dc32a2bcbd91c23673b5b511d64" exitCode=60
Nov 02 08:30:22.480366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.480289    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv" event={"ID":"eb7a2f08-f10d-4430-b0be-60892b129043","Type":"ContainerDied","Data":"f99cc76d79065b1a679405b7a03740cba6534dc32a2bcbd91c23673b5b511d64"}
Nov 02 08:30:22.480784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.480763    4898 scope.go:122] "RemoveContainer" containerID="f99cc76d79065b1a679405b7a03740cba6534dc32a2bcbd91c23673b5b511d64"
Nov 02 08:30:22.482541 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:22.481278    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv_odh-ai-gateway-infra(eb7a2f08-f10d-4430-b0be-60892b129043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv" podUID="eb7a2f08-f10d-4430-b0be-60892b129043"
Nov 02 08:30:22.485138 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.485101    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w" event={"ID":"e9d7ab63-f8af-475d-a6e8-67e7d5587603","Type":"ContainerDied","Data":"cbc8fca8f96cdefac671d0137e4a6a84c4e06112dd45e0affd46dd688ac08751"}
Nov 02 08:30:22.485352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.485198    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"
Nov 02 08:30:22.494948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.491498    4898 scope.go:122] "RemoveContainer" containerID="00969c9d50231ff091df0a5d39a43a77e4862938bd1938032a6d190603c0703e"
Nov 02 08:30:22.505001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.500482    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"
Nov 02 08:30:22.505001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.500598    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq" event={"ID":"3109b9e6-5dfe-4422-a630-b9c42c6bc1ab","Type":"ContainerDied","Data":"81b3a6111733496420ee3866c586e5df3b8634e0e7f5b185eb7731da0bbecb9e"}
Nov 02 08:30:22.509212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.509119    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq" event={"ID":"48b72892-6169-4ffa-bc4a-e64ad46688ae","Type":"ContainerStarted","Data":"0decf62232836fe15ca6c39e69a2ee718859ee18e4583ea731e6da320896de7e"}
Nov 02 08:30:22.514330 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.514297    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv" event={"ID":"85f9a6a2-8c24-4edb-a91c-b2d906ddac6f","Type":"ContainerDied","Data":"c92808ad21eb3fac6dae4a953354e0ff0e277040945133b19886a87331f688fc"}
Nov 02 08:30:22.514460 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.514417    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"
Nov 02 08:30:22.520745 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.520713    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74" event={"ID":"2abea0a3-c4f1-40df-af27-a75342569b3b","Type":"ContainerDied","Data":"d51f3fefe529ba01e7d05ca835baeb32aa7e6ec3fe39c7fb291902755f8b7414"}
Nov 02 08:30:22.520883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.520825    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"
Nov 02 08:30:22.543759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.543744    4898 scope.go:122] "RemoveContainer" containerID="cb74ef647776a45edf5d15914698ec91ed70c1b1dfbf2522cd199b166763f072"
Nov 02 08:30:22.589570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.588115    4898 scope.go:122] "RemoveContainer" containerID="00bb186e1fdaa2869194126414e6d215a526691a479aadc564cd165f3f5a2da0"
Nov 02 08:30:22.627781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.627758    4898 scope.go:122] "RemoveContainer" containerID="864acbb02babcd638a05b92e6c28473e95d3e85f71dc24b0840919fca976f389"
Nov 02 08:30:22.643788 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.643762    4898 scope.go:122] "RemoveContainer" containerID="8d23bc74e49a32153d6a86eda678d6eea96ed46c15eb9156c00695df75c2ff61"
Nov 02 08:30:22.744586 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.744551    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"
Nov 02 08:30:22.777067 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.777037    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"]
Nov 02 08:30:22.780761 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.780736    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"
Nov 02 08:30:22.782493 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.782465    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7e7a83-29893470-qnnfw"]
Nov 02 08:30:22.815070 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.815036    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"]
Nov 02 08:30:22.819993 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.819831    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-32ea65-29893470-x742w"]
Nov 02 08:30:22.853596 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.853562    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"]
Nov 02 08:30:22.882911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.878967    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-kube-api-access-56th2\" (UniqueName: \"kubernetes.io/projected/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-kube-api-access-56th2\") pod \"d85b69ee-2f64-45f1-af4e-9f23b122f2c8\" (UID: \"d85b69ee-2f64-45f1-af4e-9f23b122f2c8\") "
Nov 02 08:30:22.882911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.879053    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-maas-api-service-ca\") pod \"d85b69ee-2f64-45f1-af4e-9f23b122f2c8\" (UID: \"d85b69ee-2f64-45f1-af4e-9f23b122f2c8\") "
Nov 02 08:30:22.882911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.879512    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-maas-api-service-ca" pod "d85b69ee-2f64-45f1-af4e-9f23b122f2c8" (UID: "d85b69ee-2f64-45f1-af4e-9f23b122f2c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:22.882911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.881006    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a259b571-ddfe-400f-a22b-7f16768a82f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a259b571-ddfe-400f-a22b-7f16768a82f7-maas-api-service-ca\") pod \"a259b571-ddfe-400f-a22b-7f16768a82f7\" (UID: \"a259b571-ddfe-400f-a22b-7f16768a82f7\") "
Nov 02 08:30:22.882911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.881193    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a259b571-ddfe-400f-a22b-7f16768a82f7-kube-api-access-j7zrw\" (UniqueName: \"kubernetes.io/projected/a259b571-ddfe-400f-a22b-7f16768a82f7-kube-api-access-j7zrw\") pod \"a259b571-ddfe-400f-a22b-7f16768a82f7\" (UID: \"a259b571-ddfe-400f-a22b-7f16768a82f7\") "
Nov 02 08:30:22.882911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.881760    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a259b571-ddfe-400f-a22b-7f16768a82f7-maas-api-service-ca" pod "a259b571-ddfe-400f-a22b-7f16768a82f7" (UID: "a259b571-ddfe-400f-a22b-7f16768a82f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:22.882911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.881826    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.882911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.881824    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-kube-api-access-56th2" pod "d85b69ee-2f64-45f1-af4e-9f23b122f2c8" (UID: "d85b69ee-2f64-45f1-af4e-9f23b122f2c8"). InnerVolumeSpecName "kube-api-access-56th2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:22.884272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.884242    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a259b571-ddfe-400f-a22b-7f16768a82f7-kube-api-access-j7zrw" pod "a259b571-ddfe-400f-a22b-7f16768a82f7" (UID: "a259b571-ddfe-400f-a22b-7f16768a82f7"). InnerVolumeSpecName "kube-api-access-j7zrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:22.907220 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.907185    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"]
Nov 02 08:30:22.914410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.914377    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"]
Nov 02 08:30:22.922049 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.922020    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea9556-29893470-jb6qq"]
Nov 02 08:30:22.943372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.943341    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"]
Nov 02 08:30:22.957339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.957302    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-643355-29893470-9hp74"]
Nov 02 08:30:22.984282 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.983799    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56th2\" (UniqueName: \"kubernetes.io/projected/d85b69ee-2f64-45f1-af4e-9f23b122f2c8-kube-api-access-56th2\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.984282 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.983839    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a259b571-ddfe-400f-a22b-7f16768a82f7-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:22.984282 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:22.983858    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7zrw\" (UniqueName: \"kubernetes.io/projected/a259b571-ddfe-400f-a22b-7f16768a82f7-kube-api-access-j7zrw\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:23.001899 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.001839    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"]
Nov 02 08:30:23.011806 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.011776    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7f19e8-29893470-v96fv"]
Nov 02 08:30:23.108807 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.108780    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"
Nov 02 08:30:23.186585 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.186543    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045dedc0-929e-4caf-8349-11463c81fc48-kube-api-access-rvdvg\" (UniqueName: \"kubernetes.io/projected/045dedc0-929e-4caf-8349-11463c81fc48-kube-api-access-rvdvg\") pod \"045dedc0-929e-4caf-8349-11463c81fc48\" (UID: \"045dedc0-929e-4caf-8349-11463c81fc48\") "
Nov 02 08:30:23.186838 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.186671    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045dedc0-929e-4caf-8349-11463c81fc48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045dedc0-929e-4caf-8349-11463c81fc48-maas-api-service-ca\") pod \"045dedc0-929e-4caf-8349-11463c81fc48\" (UID: \"045dedc0-929e-4caf-8349-11463c81fc48\") "
Nov 02 08:30:23.188050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.187350    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045dedc0-929e-4caf-8349-11463c81fc48-maas-api-service-ca" pod "045dedc0-929e-4caf-8349-11463c81fc48" (UID: "045dedc0-929e-4caf-8349-11463c81fc48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:23.188050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.187951    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045dedc0-929e-4caf-8349-11463c81fc48-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:23.189946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.189915    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045dedc0-929e-4caf-8349-11463c81fc48-kube-api-access-rvdvg" pod "045dedc0-929e-4caf-8349-11463c81fc48" (UID: "045dedc0-929e-4caf-8349-11463c81fc48"). InnerVolumeSpecName "kube-api-access-rvdvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:23.289171 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.289078    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvdvg\" (UniqueName: \"kubernetes.io/projected/045dedc0-929e-4caf-8349-11463c81fc48-kube-api-access-rvdvg\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:23.471009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.470971    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b" path="/var/lib/kubelet/pods/2abea0a3-c4f1-40df-af27-a75342569b3b/volumes"
Nov 02 08:30:23.472356 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.472329    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" path="/var/lib/kubelet/pods/3109b9e6-5dfe-4422-a630-b9c42c6bc1ab/volumes"
Nov 02 08:30:23.473198 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.473177    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" path="/var/lib/kubelet/pods/6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32/volumes"
Nov 02 08:30:23.474425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.474405    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" path="/var/lib/kubelet/pods/85f9a6a2-8c24-4edb-a91c-b2d906ddac6f/volumes"
Nov 02 08:30:23.475268 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.475248    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0539990-8490-41d3-8dba-77f7e8a6af14" path="/var/lib/kubelet/pods/b0539990-8490-41d3-8dba-77f7e8a6af14/volumes"
Nov 02 08:30:23.476107 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.476087    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" path="/var/lib/kubelet/pods/e9d7ab63-f8af-475d-a6e8-67e7d5587603/volumes"
Nov 02 08:30:23.529475 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.529441    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"]
Nov 02 08:30:23.540078 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.539995    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx" event={"ID":"045dedc0-929e-4caf-8349-11463c81fc48","Type":"ContainerDied","Data":"ec5ea67753b1cea7097ac803a05e4196be2444b537b35b45a13a9d04d6a309b1"}
Nov 02 08:30:23.540078 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.540060    4898 scope.go:122] "RemoveContainer" containerID="22758c59ad1b0a7cc7a27eb77fe25ae020119ed2c43de112698d5b0095e1ab4a"
Nov 02 08:30:23.540309 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.540002    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"
Nov 02 08:30:23.544803 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.544765    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"
Nov 02 08:30:23.544969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.544775    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn" event={"ID":"d85b69ee-2f64-45f1-af4e-9f23b122f2c8","Type":"ContainerDied","Data":"b4b48b5595a98cfe8e1cccfc11f23bd22e83716f58406e533a8cd963801c189d"}
Nov 02 08:30:23.558439 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.558405    4898 generic.go:356] "Generic (PLEG): container finished" podID="48b72892-6169-4ffa-bc4a-e64ad46688ae" containerID="0decf62232836fe15ca6c39e69a2ee718859ee18e4583ea731e6da320896de7e" exitCode=60
Nov 02 08:30:23.558583 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.558474    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq" event={"ID":"48b72892-6169-4ffa-bc4a-e64ad46688ae","Type":"ContainerDied","Data":"0decf62232836fe15ca6c39e69a2ee718859ee18e4583ea731e6da320896de7e"}
Nov 02 08:30:23.558925 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.558908    4898 scope.go:122] "RemoveContainer" containerID="0decf62232836fe15ca6c39e69a2ee718859ee18e4583ea731e6da320896de7e"
Nov 02 08:30:23.559237 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:23.559215    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq_odh-ai-gateway-infra(48b72892-6169-4ffa-bc4a-e64ad46688ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae"
Nov 02 08:30:23.560735 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.560717    4898 scope.go:122] "RemoveContainer" containerID="699c8ae5634bec7aec20ecced6603cfe3df666f9cb956641ff63be6305d97e58"
Nov 02 08:30:23.563952 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.563614    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"
Nov 02 08:30:23.563952 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.563635    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp" event={"ID":"a259b571-ddfe-400f-a22b-7f16768a82f7","Type":"ContainerDied","Data":"b3ca00741947492cf04cd1fca693c51c3929dbb920dd2eb8053983dc5caf2aeb"}
Nov 02 08:30:23.566517 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.566372    4898 generic.go:356] "Generic (PLEG): container finished" podID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" containerID="9d3226124d6627098db5ca2ebf74fd5d606ed1fcae707d81aa4e08cebe1c225c" exitCode=60
Nov 02 08:30:23.566517 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.566438    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t" event={"ID":"0e09d271-5894-4f0d-9a38-dca2b8efb7bb","Type":"ContainerDied","Data":"9d3226124d6627098db5ca2ebf74fd5d606ed1fcae707d81aa4e08cebe1c225c"}
Nov 02 08:30:23.571348 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.571328    4898 generic.go:356] "Generic (PLEG): container finished" podID="eeb4d901-47b4-4247-b099-db9bd7144ce2" containerID="ecbef460d498f286f7541a68d4b113775a3d7dd558329162afd8d8eb48625d97" exitCode=60
Nov 02 08:30:23.571428 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.571360    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc" event={"ID":"eeb4d901-47b4-4247-b099-db9bd7144ce2","Type":"ContainerDied","Data":"ecbef460d498f286f7541a68d4b113775a3d7dd558329162afd8d8eb48625d97"}
Nov 02 08:30:23.571749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.571731    4898 scope.go:122] "RemoveContainer" containerID="ecbef460d498f286f7541a68d4b113775a3d7dd558329162afd8d8eb48625d97"
Nov 02 08:30:23.572127 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:23.572105    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc_odh-ai-gateway-infra(eeb4d901-47b4-4247-b099-db9bd7144ce2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2"
Nov 02 08:30:23.579136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.579109    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"]
Nov 02 08:30:23.582244 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.582224    4898 scope.go:122] "RemoveContainer" containerID="d292179741c62ea7fb23a70f335a13e2c9f07afb6f96a5deb43f819d325ede34"
Nov 02 08:30:23.597118 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.596938    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"]
Nov 02 08:30:23.606870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.606840    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb4907-29893470-2vltx"]
Nov 02 08:30:23.609145 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.608360    4898 scope.go:122] "RemoveContainer" containerID="cee73badebe2fbcf0e45fa4a577af0fa024f0405c8a4c138a80437e8eed73252"
Nov 02 08:30:23.621158 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.621135    4898 scope.go:122] "RemoveContainer" containerID="08f85ada8a057343e53589634b3520a45dc940e4d6fa5e0ead01ba2e67c9e018"
Nov 02 08:30:23.672709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.672674    4898 scope.go:122] "RemoveContainer" containerID="00dd46a535c978b4e579c6e4a2be1a61e0d8dd2fb5ffaede111acf5de3bdbbb2"
Nov 02 08:30:23.688298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.688263    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"]
Nov 02 08:30:23.692796 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.692771    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-12df3c-29893470-9dwwn"]
Nov 02 08:30:23.736696 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.736662    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"]
Nov 02 08:30:23.765221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.765189    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"
Nov 02 08:30:23.811380 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.811287    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"
Nov 02 08:30:23.848586 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:23.848551    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice/crio-66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:23.849856 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.849825    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"]
Nov 02 08:30:23.856061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.856035    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-337764-29893470-4gchp"]
Nov 02 08:30:23.897358 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.897309    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-maas-api-service-ca\") pod \"0e09d271-5894-4f0d-9a38-dca2b8efb7bb\" (UID: \"0e09d271-5894-4f0d-9a38-dca2b8efb7bb\") "
Nov 02 08:30:23.897563 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.897445    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-kube-api-access-hptf9\" (UniqueName: \"kubernetes.io/projected/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-kube-api-access-hptf9\") pod \"0e09d271-5894-4f0d-9a38-dca2b8efb7bb\" (UID: \"0e09d271-5894-4f0d-9a38-dca2b8efb7bb\") "
Nov 02 08:30:23.897563 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.897515    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb7a2f08-f10d-4430-b0be-60892b129043-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb7a2f08-f10d-4430-b0be-60892b129043-maas-api-service-ca\") pod \"eb7a2f08-f10d-4430-b0be-60892b129043\" (UID: \"eb7a2f08-f10d-4430-b0be-60892b129043\") "
Nov 02 08:30:23.897685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.897584    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7a2f08-f10d-4430-b0be-60892b129043-kube-api-access-w9dbx\" (UniqueName: \"kubernetes.io/projected/eb7a2f08-f10d-4430-b0be-60892b129043-kube-api-access-w9dbx\") pod \"eb7a2f08-f10d-4430-b0be-60892b129043\" (UID: \"eb7a2f08-f10d-4430-b0be-60892b129043\") "
Nov 02 08:30:23.897748 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.897715    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-maas-api-service-ca" pod "0e09d271-5894-4f0d-9a38-dca2b8efb7bb" (UID: "0e09d271-5894-4f0d-9a38-dca2b8efb7bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:23.898242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.898217    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:23.898391 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.898307    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7a2f08-f10d-4430-b0be-60892b129043-maas-api-service-ca" pod "eb7a2f08-f10d-4430-b0be-60892b129043" (UID: "eb7a2f08-f10d-4430-b0be-60892b129043"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:23.900935 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.900905    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-kube-api-access-hptf9" pod "0e09d271-5894-4f0d-9a38-dca2b8efb7bb" (UID: "0e09d271-5894-4f0d-9a38-dca2b8efb7bb"). InnerVolumeSpecName "kube-api-access-hptf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:23.901187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.901165    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7a2f08-f10d-4430-b0be-60892b129043-kube-api-access-w9dbx" pod "eb7a2f08-f10d-4430-b0be-60892b129043" (UID: "eb7a2f08-f10d-4430-b0be-60892b129043"). InnerVolumeSpecName "kube-api-access-w9dbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:23.999536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.999498    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hptf9\" (UniqueName: \"kubernetes.io/projected/0e09d271-5894-4f0d-9a38-dca2b8efb7bb-kube-api-access-hptf9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:23.999536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.999532    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb7a2f08-f10d-4430-b0be-60892b129043-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:23.999536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:23.999541    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9dbx\" (UniqueName: \"kubernetes.io/projected/eb7a2f08-f10d-4430-b0be-60892b129043-kube-api-access-w9dbx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:24.517916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.517840    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8" podUID="5f970daf-e1a2-4859-8446-328a1d214d2e" containerName="payload-processing" containerID="cri-o://eb51d89e7f50ea3f684782227381943f1742f7fddfb56f45dc3a9e1c415aae84" gracePeriod=25
Nov 02 08:30:24.548191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.548140    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk" podUID="ac230396-f0f0-418f-bba6-81b0d8d28026" containerName="payload-pre-processing" containerID="cri-o://f40c30546ccfc7aabaab1ed41881442ba275713af9e294ffd78e8238dcf76da1" gracePeriod=25
Nov 02 08:30:24.589117 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.589084    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t" event={"ID":"0e09d271-5894-4f0d-9a38-dca2b8efb7bb","Type":"ContainerDied","Data":"3449c09075ce18744caa6da534f04240023a98b4d5fc4b62747f9ef39bc441ec"}
Nov 02 08:30:24.589281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.589093    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"
Nov 02 08:30:24.589281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.589152    4898 scope.go:122] "RemoveContainer" containerID="9d3226124d6627098db5ca2ebf74fd5d606ed1fcae707d81aa4e08cebe1c225c"
Nov 02 08:30:24.595030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.595003    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"
Nov 02 08:30:24.595163 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.595013    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv" event={"ID":"eb7a2f08-f10d-4430-b0be-60892b129043","Type":"ContainerDied","Data":"da2c33e8b34fa35afe7a6fa36517c9e2b716526c2261bd094aa8474e111db515"}
Nov 02 08:30:24.606732 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.606705    4898 scope.go:122] "RemoveContainer" containerID="f99cc76d79065b1a679405b7a03740cba6534dc32a2bcbd91c23673b5b511d64"
Nov 02 08:30:24.713026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.712743    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"]
Nov 02 08:30:24.717949 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.717920    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e4c131-29893470-98wpv"]
Nov 02 08:30:24.751758 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.751710    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"]
Nov 02 08:30:24.758229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.758151    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-71a41c-29893470-74g7t"]
Nov 02 08:30:24.824314 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.824282    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"
Nov 02 08:30:24.827242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.827218    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"
Nov 02 08:30:24.912955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.912920    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b72892-6169-4ffa-bc4a-e64ad46688ae-kube-api-access-cv465\" (UniqueName: \"kubernetes.io/projected/48b72892-6169-4ffa-bc4a-e64ad46688ae-kube-api-access-cv465\") pod \"48b72892-6169-4ffa-bc4a-e64ad46688ae\" (UID: \"48b72892-6169-4ffa-bc4a-e64ad46688ae\") "
Nov 02 08:30:24.913856 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.913060    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeb4d901-47b4-4247-b099-db9bd7144ce2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4d901-47b4-4247-b099-db9bd7144ce2-maas-api-service-ca\") pod \"eeb4d901-47b4-4247-b099-db9bd7144ce2\" (UID: \"eeb4d901-47b4-4247-b099-db9bd7144ce2\") "
Nov 02 08:30:24.913856 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.913136    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb4d901-47b4-4247-b099-db9bd7144ce2-kube-api-access-4pdmm\" (UniqueName: \"kubernetes.io/projected/eeb4d901-47b4-4247-b099-db9bd7144ce2-kube-api-access-4pdmm\") pod \"eeb4d901-47b4-4247-b099-db9bd7144ce2\" (UID: \"eeb4d901-47b4-4247-b099-db9bd7144ce2\") "
Nov 02 08:30:24.913856 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.913308    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b72892-6169-4ffa-bc4a-e64ad46688ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b72892-6169-4ffa-bc4a-e64ad46688ae-maas-api-service-ca\") pod \"48b72892-6169-4ffa-bc4a-e64ad46688ae\" (UID: \"48b72892-6169-4ffa-bc4a-e64ad46688ae\") "
Nov 02 08:30:24.914295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.914253    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb4d901-47b4-4247-b099-db9bd7144ce2-maas-api-service-ca" pod "eeb4d901-47b4-4247-b099-db9bd7144ce2" (UID: "eeb4d901-47b4-4247-b099-db9bd7144ce2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:24.914460 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.914437    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b72892-6169-4ffa-bc4a-e64ad46688ae-maas-api-service-ca" pod "48b72892-6169-4ffa-bc4a-e64ad46688ae" (UID: "48b72892-6169-4ffa-bc4a-e64ad46688ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:24.916746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.916718    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb4d901-47b4-4247-b099-db9bd7144ce2-kube-api-access-4pdmm" pod "eeb4d901-47b4-4247-b099-db9bd7144ce2" (UID: "eeb4d901-47b4-4247-b099-db9bd7144ce2"). InnerVolumeSpecName "kube-api-access-4pdmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:24.917141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:24.917096    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b72892-6169-4ffa-bc4a-e64ad46688ae-kube-api-access-cv465" pod "48b72892-6169-4ffa-bc4a-e64ad46688ae" (UID: "48b72892-6169-4ffa-bc4a-e64ad46688ae"). InnerVolumeSpecName "kube-api-access-cv465". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:25.015432 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.015396    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4d901-47b4-4247-b099-db9bd7144ce2-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:25.015432 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.015428    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pdmm\" (UniqueName: \"kubernetes.io/projected/eeb4d901-47b4-4247-b099-db9bd7144ce2-kube-api-access-4pdmm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:25.015432 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.015439    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b72892-6169-4ffa-bc4a-e64ad46688ae-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:25.015706 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.015448    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv465\" (UniqueName: \"kubernetes.io/projected/48b72892-6169-4ffa-bc4a-e64ad46688ae-kube-api-access-cv465\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:25.398078 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.398034    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8" podUID="5f970daf-e1a2-4859-8446-328a1d214d2e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:30:25.473385 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.473344    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045dedc0-929e-4caf-8349-11463c81fc48" path="/var/lib/kubelet/pods/045dedc0-929e-4caf-8349-11463c81fc48/volumes"
Nov 02 08:30:25.473997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.473973    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" path="/var/lib/kubelet/pods/0e09d271-5894-4f0d-9a38-dca2b8efb7bb/volumes"
Nov 02 08:30:25.474504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.474483    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7" path="/var/lib/kubelet/pods/a259b571-ddfe-400f-a22b-7f16768a82f7/volumes"
Nov 02 08:30:25.475412 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.475387    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" path="/var/lib/kubelet/pods/d85b69ee-2f64-45f1-af4e-9f23b122f2c8/volumes"
Nov 02 08:30:25.475863 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.475843    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb7a2f08-f10d-4430-b0be-60892b129043" path="/var/lib/kubelet/pods/eb7a2f08-f10d-4430-b0be-60892b129043/volumes"
Nov 02 08:30:25.612438 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.612397    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"
Nov 02 08:30:25.612438 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.612419    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq" event={"ID":"48b72892-6169-4ffa-bc4a-e64ad46688ae","Type":"ContainerDied","Data":"51571d10769cc1a13bb7392e123b45f42c953f04d326c8a40a5aba24cd29e838"}
Nov 02 08:30:25.613025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.612484    4898 scope.go:122] "RemoveContainer" containerID="0decf62232836fe15ca6c39e69a2ee718859ee18e4583ea731e6da320896de7e"
Nov 02 08:30:25.616240 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.616209    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc" event={"ID":"eeb4d901-47b4-4247-b099-db9bd7144ce2","Type":"ContainerDied","Data":"11dda57575111a1ffbf561ab334751cf384c48d9552b695935adbe02c641ae7e"}
Nov 02 08:30:25.616377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.616278    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"
Nov 02 08:30:25.626495 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.626469    4898 scope.go:122] "RemoveContainer" containerID="ecbef460d498f286f7541a68d4b113775a3d7dd558329162afd8d8eb48625d97"
Nov 02 08:30:25.669795 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.669760    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"]
Nov 02 08:30:25.673791 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.673766    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9877ce-29893470-m9ztq"]
Nov 02 08:30:25.705960 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.705929    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"]
Nov 02 08:30:25.709933 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:25.709909    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3575b9-29893470-2bspc"]
Nov 02 08:30:27.470859 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:27.470817    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae" path="/var/lib/kubelet/pods/48b72892-6169-4ffa-bc4a-e64ad46688ae/volumes"
Nov 02 08:30:27.471502 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:27.471480    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2" path="/var/lib/kubelet/pods/eeb4d901-47b4-4247-b099-db9bd7144ce2/volumes"
Nov 02 08:30:27.502686 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:27.502648    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice/crio-66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:29.047745 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:29.047697    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"]
Nov 02 08:30:29.067241 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:29.067195    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125cda00-jv5jm"]
Nov 02 08:30:29.230777 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:29.230735    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice/crio-66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:29.230979 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:29.230845    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice/crio-66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:29.467146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:29.467108    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c380a91-54c4-4d33-ba17-c5d82470bcaf" path="/var/lib/kubelet/pods/6c380a91-54c4-4d33-ba17-c5d82470bcaf/volumes"
Nov 02 08:30:30.755449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:30.755405    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt" podUID="5379dd80-6ad8-4b2a-a49c-e012dfc0fab6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:30:31.056754 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:31.056663    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"]
Nov 02 08:30:31.068220 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:31.068181    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0208019-bbf4w"]
Nov 02 08:30:31.467334 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:31.467296    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d2c733-9831-4afe-83d5-9df8920db621" path="/var/lib/kubelet/pods/99d2c733-9831-4afe-83d5-9df8920db621/volumes"
Nov 02 08:30:33.375726 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:33.375676    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk" podUID="ac230396-f0f0-418f-bba6-81b0d8d28026" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:30:33.869520 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:33.869479    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice/crio-66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:34.196483 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.196450    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"
Nov 02 08:30:34.246861 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.246828    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-kube-api-access-8dhhh\" (UniqueName: \"kubernetes.io/projected/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-kube-api-access-8dhhh\") pod \"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6\" (UID: \"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6\") "
Nov 02 08:30:34.247072 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.246881    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-plugins-config-volume\") pod \"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6\" (UID: \"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6\") "
Nov 02 08:30:34.247686 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.247652    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-plugins-config-volume" pod "5379dd80-6ad8-4b2a-a49c-e012dfc0fab6" (UID: "5379dd80-6ad8-4b2a-a49c-e012dfc0fab6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:34.249514 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.249485    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-kube-api-access-8dhhh" pod "5379dd80-6ad8-4b2a-a49c-e012dfc0fab6" (UID: "5379dd80-6ad8-4b2a-a49c-e012dfc0fab6"). InnerVolumeSpecName "kube-api-access-8dhhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:34.348743 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.348704    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dhhh\" (UniqueName: \"kubernetes.io/projected/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-kube-api-access-8dhhh\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:34.348743 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.348739    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:34.706796 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.706760    4898 generic.go:356] "Generic (PLEG): container finished" podID="5379dd80-6ad8-4b2a-a49c-e012dfc0fab6" containerID="e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d" exitCode=0
Nov 02 08:30:34.707269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.706832    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"
Nov 02 08:30:34.707269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.706850    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt" event={"ID":"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6","Type":"ContainerDied","Data":"e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d"}
Nov 02 08:30:34.707269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.706956    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt" event={"ID":"5379dd80-6ad8-4b2a-a49c-e012dfc0fab6","Type":"ContainerDied","Data":"76c2a111b72f2ee247a4cc673962240b5fc210afe311fb04dbd4e74c7d75177a"}
Nov 02 08:30:34.707269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.706983    4898 scope.go:122] "RemoveContainer" containerID="e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d"
Nov 02 08:30:34.719449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.719431    4898 scope.go:122] "RemoveContainer" containerID="e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d"
Nov 02 08:30:34.719719 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:34.719697    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d\": container with ID starting with e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d not found: ID does not exist" containerID="e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d"
Nov 02 08:30:34.719772 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.719727    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d"} err="failed to get container status \"e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d\": rpc error: code = NotFound desc = could not find container \"e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d\": container with ID starting with e0291c96ceb0e53a53e4c0d584ed9e84a32dc260f97e654c1d06f89d79d7640d not found: ID does not exist"
Nov 02 08:30:34.760781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.760740    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"]
Nov 02 08:30:34.765611 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:34.765578    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-32ea65-945f7c88c-bf9kt"]
Nov 02 08:30:35.398371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:35.398331    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8" podUID="5f970daf-e1a2-4859-8446-328a1d214d2e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:30:35.466195 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:35.466157    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5379dd80-6ad8-4b2a-a49c-e012dfc0fab6" path="/var/lib/kubelet/pods/5379dd80-6ad8-4b2a-a49c-e012dfc0fab6/volumes"
Nov 02 08:30:40.033812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:40.033774    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"]
Nov 02 08:30:40.045534 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:40.045503    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3859bc94-h5djk"]
Nov 02 08:30:41.467975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:41.467932    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750d62da-25d2-4734-90ff-ce7976d45e42" path="/var/lib/kubelet/pods/750d62da-25d2-4734-90ff-ce7976d45e42/volumes"
Nov 02 08:30:42.658184 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:42.658141    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice/crio-66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:43.375748 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:43.375708    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk" podUID="ac230396-f0f0-418f-bba6-81b0d8d28026" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:30:43.885444 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:43.885405    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice/crio-66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:44.842417 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:44.842331    4898 generic.go:356] "Generic (PLEG): container finished" podID="5f970daf-e1a2-4859-8446-328a1d214d2e" containerID="eb51d89e7f50ea3f684782227381943f1742f7fddfb56f45dc3a9e1c415aae84" exitCode=0
Nov 02 08:30:44.842644 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:44.842618    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8" event={"ID":"5f970daf-e1a2-4859-8446-328a1d214d2e","Type":"ContainerDied","Data":"eb51d89e7f50ea3f684782227381943f1742f7fddfb56f45dc3a9e1c415aae84"}
Nov 02 08:30:44.845919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:44.845878    4898 generic.go:356] "Generic (PLEG): container finished" podID="ac230396-f0f0-418f-bba6-81b0d8d28026" containerID="f40c30546ccfc7aabaab1ed41881442ba275713af9e294ffd78e8238dcf76da1" exitCode=0
Nov 02 08:30:44.846065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:44.845949    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk" event={"ID":"ac230396-f0f0-418f-bba6-81b0d8d28026","Type":"ContainerDied","Data":"f40c30546ccfc7aabaab1ed41881442ba275713af9e294ffd78e8238dcf76da1"}
Nov 02 08:30:44.907470 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:44.907436    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"
Nov 02 08:30:44.958222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:44.958190    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"
Nov 02 08:30:45.001601 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:45.000596    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:30:45.033811 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.033778    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac230396-f0f0-418f-bba6-81b0d8d28026-kube-api-access-mxph5\" (UniqueName: \"kubernetes.io/projected/ac230396-f0f0-418f-bba6-81b0d8d28026-kube-api-access-mxph5\") pod \"ac230396-f0f0-418f-bba6-81b0d8d28026\" (UID: \"ac230396-f0f0-418f-bba6-81b0d8d28026\") "
Nov 02 08:30:45.034029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.033870    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f970daf-e1a2-4859-8446-328a1d214d2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f970daf-e1a2-4859-8446-328a1d214d2e-plugins-config-volume\") pod \"5f970daf-e1a2-4859-8446-328a1d214d2e\" (UID: \"5f970daf-e1a2-4859-8446-328a1d214d2e\") "
Nov 02 08:30:45.034029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.033921    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f970daf-e1a2-4859-8446-328a1d214d2e-kube-api-access-kk6c6\" (UniqueName: \"kubernetes.io/projected/5f970daf-e1a2-4859-8446-328a1d214d2e-kube-api-access-kk6c6\") pod \"5f970daf-e1a2-4859-8446-328a1d214d2e\" (UID: \"5f970daf-e1a2-4859-8446-328a1d214d2e\") "
Nov 02 08:30:45.034029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.033966    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac230396-f0f0-418f-bba6-81b0d8d28026-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac230396-f0f0-418f-bba6-81b0d8d28026-plugins-config-volume\") pod \"ac230396-f0f0-418f-bba6-81b0d8d28026\" (UID: \"ac230396-f0f0-418f-bba6-81b0d8d28026\") "
Nov 02 08:30:45.034372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.034324    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f970daf-e1a2-4859-8446-328a1d214d2e-plugins-config-volume" pod "5f970daf-e1a2-4859-8446-328a1d214d2e" (UID: "5f970daf-e1a2-4859-8446-328a1d214d2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:45.034471 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.034417    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac230396-f0f0-418f-bba6-81b0d8d28026-plugins-config-volume" pod "ac230396-f0f0-418f-bba6-81b0d8d28026" (UID: "ac230396-f0f0-418f-bba6-81b0d8d28026"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:45.034788 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.034765    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f970daf-e1a2-4859-8446-328a1d214d2e-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:45.034848 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.034796    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac230396-f0f0-418f-bba6-81b0d8d28026-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:45.036171 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.036141    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac230396-f0f0-418f-bba6-81b0d8d28026-kube-api-access-mxph5" pod "ac230396-f0f0-418f-bba6-81b0d8d28026" (UID: "ac230396-f0f0-418f-bba6-81b0d8d28026"). InnerVolumeSpecName "kube-api-access-mxph5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:45.036579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.036560    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f970daf-e1a2-4859-8446-328a1d214d2e-kube-api-access-kk6c6" pod "5f970daf-e1a2-4859-8446-328a1d214d2e" (UID: "5f970daf-e1a2-4859-8446-328a1d214d2e"). InnerVolumeSpecName "kube-api-access-kk6c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:45.135983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.135875    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxph5\" (UniqueName: \"kubernetes.io/projected/ac230396-f0f0-418f-bba6-81b0d8d28026-kube-api-access-mxph5\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:45.135983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.135937    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk6c6\" (UniqueName: \"kubernetes.io/projected/5f970daf-e1a2-4859-8446-328a1d214d2e-kube-api-access-kk6c6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:45.855215 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.855131    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"
Nov 02 08:30:45.855215 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.855165    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk" event={"ID":"ac230396-f0f0-418f-bba6-81b0d8d28026","Type":"ContainerDied","Data":"898434b7399216e6097a53bab7f54ca07c8e227ffd8bbd950cb6c1313e101b80"}
Nov 02 08:30:45.855462 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.855236    4898 scope.go:122] "RemoveContainer" containerID="f40c30546ccfc7aabaab1ed41881442ba275713af9e294ffd78e8238dcf76da1"
Nov 02 08:30:45.857377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.857351    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8" event={"ID":"5f970daf-e1a2-4859-8446-328a1d214d2e","Type":"ContainerDied","Data":"1292c8c89a1bfd218927e766acd81ba1411fd37eb2fcbc865a7c30351665ccbd"}
Nov 02 08:30:45.857518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.857442    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"
Nov 02 08:30:45.868658 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.868638    4898 scope.go:122] "RemoveContainer" containerID="eb51d89e7f50ea3f684782227381943f1742f7fddfb56f45dc3a9e1c415aae84"
Nov 02 08:30:45.912544 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.912519    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"]
Nov 02 08:30:45.926293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.926267    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2365fa-7f5b898d87-f95rk"]
Nov 02 08:30:45.953093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.953060    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"]
Nov 02 08:30:45.956774 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:45.956745    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2365fa-798d564587-vkrp8"]
Nov 02 08:30:47.031133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:47.031097    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"]
Nov 02 08:30:47.045067 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:47.045036    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e574e612-gvv54"]
Nov 02 08:30:47.466900 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:47.466850    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04cdcd7c-7bd5-48e6-885e-29d3b8049bf1" path="/var/lib/kubelet/pods/04cdcd7c-7bd5-48e6-885e-29d3b8049bf1/volumes"
Nov 02 08:30:47.467376 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:47.467359    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f970daf-e1a2-4859-8446-328a1d214d2e" path="/var/lib/kubelet/pods/5f970daf-e1a2-4859-8446-328a1d214d2e/volumes"
Nov 02 08:30:47.467743 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:47.467730    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac230396-f0f0-418f-bba6-81b0d8d28026" path="/var/lib/kubelet/pods/ac230396-f0f0-418f-bba6-81b0d8d28026/volumes"
Nov 02 08:30:53.074033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:53.073932    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"]
Nov 02 08:30:53.074523 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:53.074236    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy" containerID="cri-o://62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c" gracePeriod=30
Nov 02 08:30:53.901741 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:53.901698    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice/crio-66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:57.575339 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:57.575296    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice/crio-66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 08:30:58.374657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.374624    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 08:30:58.392805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.392774    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-kube-api-access-4g2ds\" (UniqueName: \"kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-kube-api-access-4g2ds\") pod \"2c33009d-0656-4873-9be6-968f615c990d\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") "
Nov 02 08:30:58.393003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.392838    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-socket\") pod \"2c33009d-0656-4873-9be6-968f615c990d\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") "
Nov 02 08:30:58.393003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.392862    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-istio-token\" (UniqueName: \"kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-istio-token\") pod \"2c33009d-0656-4873-9be6-968f615c990d\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") "
Nov 02 08:30:58.393003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.392883    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-certs\") pod \"2c33009d-0656-4873-9be6-968f615c990d\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") "
Nov 02 08:30:58.393003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.392965    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c33009d-0656-4873-9be6-968f615c990d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c33009d-0656-4873-9be6-968f615c990d-istio-podinfo\") pod \"2c33009d-0656-4873-9be6-968f615c990d\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") "
Nov 02 08:30:58.393003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.392997    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-credential-socket\") pod \"2c33009d-0656-4873-9be6-968f615c990d\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") "
Nov 02 08:30:58.393305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393057    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-data\") pod \"2c33009d-0656-4873-9be6-968f615c990d\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") "
Nov 02 08:30:58.393305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393120    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c33009d-0656-4873-9be6-968f615c990d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c33009d-0656-4873-9be6-968f615c990d-istiod-ca-cert\") pod \"2c33009d-0656-4873-9be6-968f615c990d\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") "
Nov 02 08:30:58.393305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393231    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-envoy\") pod \"2c33009d-0656-4873-9be6-968f615c990d\" (UID: \"2c33009d-0656-4873-9be6-968f615c990d\") "
Nov 02 08:30:58.393305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393220    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-socket" pod "2c33009d-0656-4873-9be6-968f615c990d" (UID: "2c33009d-0656-4873-9be6-968f615c990d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:58.393549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393296    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-certs" pod "2c33009d-0656-4873-9be6-968f615c990d" (UID: "2c33009d-0656-4873-9be6-968f615c990d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:58.393549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393379    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-credential-socket" pod "2c33009d-0656-4873-9be6-968f615c990d" (UID: "2c33009d-0656-4873-9be6-968f615c990d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:58.393549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393430    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-data" pod "2c33009d-0656-4873-9be6-968f615c990d" (UID: "2c33009d-0656-4873-9be6-968f615c990d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:58.393717 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393602    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c33009d-0656-4873-9be6-968f615c990d-istiod-ca-cert" pod "2c33009d-0656-4873-9be6-968f615c990d" (UID: "2c33009d-0656-4873-9be6-968f615c990d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:30:58.393847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393825    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:58.393847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393847    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:58.394004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393856    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:58.394004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393865    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:58.394004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.393874    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c33009d-0656-4873-9be6-968f615c990d-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:58.395507 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.395480    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-envoy" pod "2c33009d-0656-4873-9be6-968f615c990d" (UID: "2c33009d-0656-4873-9be6-968f615c990d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:30:58.395722 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.395703    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-istio-token" pod "2c33009d-0656-4873-9be6-968f615c990d" (UID: "2c33009d-0656-4873-9be6-968f615c990d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:58.395807 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.395720    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c33009d-0656-4873-9be6-968f615c990d-istio-podinfo" pod "2c33009d-0656-4873-9be6-968f615c990d" (UID: "2c33009d-0656-4873-9be6-968f615c990d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:30:58.395807 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.395733    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-kube-api-access-4g2ds" pod "2c33009d-0656-4873-9be6-968f615c990d" (UID: "2c33009d-0656-4873-9be6-968f615c990d"). InnerVolumeSpecName "kube-api-access-4g2ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:30:58.495134 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.495094    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c33009d-0656-4873-9be6-968f615c990d-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:58.495134 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.495134    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g2ds\" (UniqueName: \"kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-kube-api-access-4g2ds\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:58.495354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.495149    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c33009d-0656-4873-9be6-968f615c990d-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:58.495354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:58.495164    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c33009d-0656-4873-9be6-968f615c990d-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:30:59.000335 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.000296    4898 generic.go:356] "Generic (PLEG): container finished" podID="2c33009d-0656-4873-9be6-968f615c990d" containerID="62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c" exitCode=0
Nov 02 08:30:59.000803 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.000364    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"
Nov 02 08:30:59.000803 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.000387    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" event={"ID":"2c33009d-0656-4873-9be6-968f615c990d","Type":"ContainerDied","Data":"62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c"}
Nov 02 08:30:59.000803 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.000448    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb" event={"ID":"2c33009d-0656-4873-9be6-968f615c990d","Type":"ContainerDied","Data":"d0992753ec3ca6d8e5319324cfe2d363441274e23a18782030b36258084ca5df"}
Nov 02 08:30:59.000803 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.000476    4898 scope.go:122] "RemoveContainer" containerID="62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c"
Nov 02 08:30:59.014429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.014403    4898 scope.go:122] "RemoveContainer" containerID="897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b"
Nov 02 08:30:59.025609 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.025588    4898 scope.go:122] "RemoveContainer" containerID="62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c"
Nov 02 08:30:59.025920 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:59.025873    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c\": container with ID starting with 62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c not found: ID does not exist" containerID="62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c"
Nov 02 08:30:59.025999 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.025919    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c"} err="failed to get container status \"62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c\": rpc error: code = NotFound desc = could not find container \"62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c\": container with ID starting with 62205decbe81bb046cf77707b9d864baaa95691dfac052b8b4926ef5fbb7ea2c not found: ID does not exist"
Nov 02 08:30:59.025999 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.025942    4898 scope.go:122] "RemoveContainer" containerID="897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b"
Nov 02 08:30:59.026205 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:30:59.026187    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b\": container with ID starting with 897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b not found: ID does not exist" containerID="897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b"
Nov 02 08:30:59.026248 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.026213    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b"} err="failed to get container status \"897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b\": rpc error: code = NotFound desc = could not find container \"897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b\": container with ID starting with 897ad449a1191bb6dcb388914d90b9587d3688c402f58fdd2fa04e7a7396e36b not found: ID does not exist"
Nov 02 08:30:59.069245 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.069210    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"]
Nov 02 08:30:59.081474 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.081442    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-32ea65-openshift-default-595544d6dc-7dsfb"]
Nov 02 08:30:59.466564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:30:59.466530    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c33009d-0656-4873-9be6-968f615c990d" path="/var/lib/kubelet/pods/2c33009d-0656-4873-9be6-968f615c990d/volumes"
Nov 02 08:31:00.385423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:00.385384    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"]
Nov 02 08:31:03.987179 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:31:03.987140    4898 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8398fcd_e412_4abe_b630_78573f4cf7fe.slice/crio-66dc16f3b7ac9e18526d375e787e342fe0bfe98ace44d866f849f52c95715e33\": RecentStats: unable to find data in memory cache]"
Nov 02 08:31:05.386525 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:05.386440    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj" podUID="13a7ae74-c294-4de1-b499-a2e0a7318cfc" containerName="payload-processing" containerID="cri-o://ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d" gracePeriod=25
Nov 02 08:31:05.822766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:05.822727    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj" podUID="13a7ae74-c294-4de1-b499-a2e0a7318cfc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:31:08.709823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.709772    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"]
Nov 02 08:31:08.710780 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710755    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710784    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710800    4898 cpu_manager.go:414] "containerMap: removing container" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710808    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710825    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b0539990-8490-41d3-8dba-77f7e8a6af14" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710833    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0539990-8490-41d3-8dba-77f7e8a6af14" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710847    4898 cpu_manager.go:414] "containerMap: removing container" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710856    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710867    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710874    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710925    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b0539990-8490-41d3-8dba-77f7e8a6af14" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710934    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0539990-8490-41d3-8dba-77f7e8a6af14" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710945    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710952    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710961    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710968    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710981    4898 cpu_manager.go:414] "containerMap: removing container" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae" containerName="cleanup"
Nov 02 08:31:08.710983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710989    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.710999    4898 cpu_manager.go:414] "containerMap: removing container" podUID="045dedc0-929e-4caf-8349-11463c81fc48" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711008    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045dedc0-929e-4caf-8349-11463c81fc48" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711021    4898 cpu_manager.go:414] "containerMap: removing container" podUID="5379dd80-6ad8-4b2a-a49c-e012dfc0fab6" containerName="payload-pre-processing"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711029    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5379dd80-6ad8-4b2a-a49c-e012dfc0fab6" containerName="payload-pre-processing"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711039    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711047    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711057    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711064    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711077    4898 cpu_manager.go:414] "containerMap: removing container" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711084    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711094    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711101    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711113    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711120    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711134    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711141    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711152    4898 cpu_manager.go:414] "containerMap: removing container" podUID="eb7a2f08-f10d-4430-b0be-60892b129043" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711160    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7a2f08-f10d-4430-b0be-60892b129043" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711172    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711179    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711190    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711196    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711216    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d8398fcd-e412-4abe-b630-78573f4cf7fe" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711225    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8398fcd-e412-4abe-b630-78573f4cf7fe" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711240    4898 cpu_manager.go:414] "containerMap: removing container" podUID="045dedc0-929e-4caf-8349-11463c81fc48" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711247    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045dedc0-929e-4caf-8349-11463c81fc48" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711256    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerName="istio-proxy"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711263    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerName="istio-proxy"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711273    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b0539990-8490-41d3-8dba-77f7e8a6af14" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711282    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0539990-8490-41d3-8dba-77f7e8a6af14" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711297    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711305    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711318    4898 cpu_manager.go:414] "containerMap: removing container" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711325    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711334    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711340    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711353    4898 cpu_manager.go:414] "containerMap: removing container" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711360    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711370    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711377    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711385    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711392    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711446    4898 cpu_manager.go:414] "containerMap: removing container" podUID="5f970daf-e1a2-4859-8446-328a1d214d2e" containerName="payload-processing"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711455    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f970daf-e1a2-4859-8446-328a1d214d2e" containerName="payload-processing"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711468    4898 cpu_manager.go:414] "containerMap: removing container" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711475    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711486    4898 cpu_manager.go:414] "containerMap: removing container" podUID="eb7a2f08-f10d-4430-b0be-60892b129043" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711493    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7a2f08-f10d-4430-b0be-60892b129043" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711516    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ac230396-f0f0-418f-bba6-81b0d8d28026" containerName="payload-pre-processing"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711523    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac230396-f0f0-418f-bba6-81b0d8d28026" containerName="payload-pre-processing"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711535    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711543    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711551    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7" containerName="cleanup"
Nov 02 08:31:08.711550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711558    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711579    4898 cpu_manager.go:414] "containerMap: removing container" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711586    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711600    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711607    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711767    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711784    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="045dedc0-929e-4caf-8349-11463c81fc48" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711793    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711801    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711811    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711819    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711827    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711835    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0539990-8490-41d3-8dba-77f7e8a6af14" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711842    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711849    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711859    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0539990-8490-41d3-8dba-77f7e8a6af14" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711866    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711875    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711903    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711913    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="045dedc0-929e-4caf-8349-11463c81fc48" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711921    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5379dd80-6ad8-4b2a-a49c-e012dfc0fab6" containerName="payload-pre-processing"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711930    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711938    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711948    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711957    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711967    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711977    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711985    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7a2f08-f10d-4430-b0be-60892b129043" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.711992    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712000    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712012    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8398fcd-e412-4abe-b630-78573f4cf7fe" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712022    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerName="istio-proxy"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712031    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712040    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712055    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712065    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712071    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b2f1d7-fe94-49ea-89d9-6c6c02f73bfc" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712085    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712094    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f970daf-e1a2-4859-8446-328a1d214d2e" containerName="payload-processing"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712104    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3b67b4-66e9-4f49-942d-61b4b632ac82" containerName="istio-proxy"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712115    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7a2f08-f10d-4430-b0be-60892b129043" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712124    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7a2f08-f10d-4430-b0be-60892b129043" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712135    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712145    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712164    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8398fcd-e412-4abe-b630-78573f4cf7fe" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712177    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712186    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac230396-f0f0-418f-bba6-81b0d8d28026" containerName="payload-pre-processing"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712197    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" containerName="cleanup"
Nov 02 08:31:08.713581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.712204    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7" containerName="cleanup"
Nov 02 08:31:08.716320 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.716296    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.734200 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.734168    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-3d8293-openshift-default-dockercfg-zmg2l\""
Nov 02 08:31:08.736296 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.736264    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"]
Nov 02 08:31:08.787963 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.787923    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-credential-socket\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.788160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.787967    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-token\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.788160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.788061    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-certs\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.788258 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.788227    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-socket\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.788313 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.788300    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfhgw\" (UniqueName: \"kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-kube-api-access-qfhgw\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.788354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.788329    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-data\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.788440 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.788421    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-envoy\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.788572 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.788552    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-podinfo\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.788613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.788584    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istiod-ca-cert\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.889406 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.889364    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-token\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.889599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.889429    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-certs\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.889599 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.889541    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-socket\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.889739 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.889617    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfhgw\" (UniqueName: \"kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-kube-api-access-qfhgw\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.890365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.889654    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-data\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.890365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.889965    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-certs\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.890365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.889995    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-envoy\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.890365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.890096    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-socket\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.890365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.890115    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-podinfo\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.890365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.890146    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istiod-ca-cert\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.890365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.890175    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-data\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.890365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.890195    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-credential-socket\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.890956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.890491    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-credential-socket\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.891107 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.891082    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istiod-ca-cert\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.892869 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.892843    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-envoy\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.893020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.892849    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-podinfo\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.941914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.927560    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-token\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:08.954281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:08.954208    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfhgw\" (UniqueName: \"kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-kube-api-access-qfhgw\") pod \"e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") " pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:09.036286 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:09.036181    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:09.623192 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:09.622654    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"]
Nov 02 08:31:09.630835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:09.630798    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:31:09.630972 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:09.630868    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:31:09.630972 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:09.630922    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:31:10.124492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:10.124454    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" event={"ID":"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f","Type":"ContainerStarted","Data":"c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973"}
Nov 02 08:31:10.125016 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:10.124512    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" event={"ID":"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f","Type":"ContainerStarted","Data":"4da0c2bfc699ef1ac8df596ef6670a84f50e992ed5b67693ea74bb7f322b0f9e"}
Nov 02 08:31:10.188307 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:10.188254    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" podStartSLOduration=2.1882399 podStartE2EDuration="2.1882399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:10.183876097 +0000 UTC m=+4020.874203064" watchObservedRunningTime="2026-11-02 08:31:10.1882399 +0000 UTC m=+4020.878566863"
Nov 02 08:31:11.037619 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:11.037585    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:11.043461 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:11.043433    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:11.133414 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:11.133378    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:11.134568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:11.134546    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:31:12.050261 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:12.050219    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"]
Nov 02 08:31:12.062133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:12.062096    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9006596f-9c5wq"]
Nov 02 08:31:13.467273 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:13.467238    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4551b1b-251b-4e79-b202-895272200c23" path="/var/lib/kubelet/pods/a4551b1b-251b-4e79-b202-895272200c23/volumes"
Nov 02 08:31:15.018615 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:31:15.018565    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:31:15.055281 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:15.055240    4898 scope.go:122] "RemoveContainer" containerID="25b73451ab794f5cac100ff54c5ac52943f4f974201ddd105c5a5ab52f0c6261"
Nov 02 08:31:15.825939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:15.823271    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj" podUID="13a7ae74-c294-4de1-b499-a2e0a7318cfc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:31:25.035121 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:25.035082    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"]
Nov 02 08:31:25.044606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:25.044575    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff60616-99rwd"]
Nov 02 08:31:25.467671 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:25.467634    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43" path="/var/lib/kubelet/pods/ed8b0a1d-1a47-4014-8f56-e13d9cd4bb43/volumes"
Nov 02 08:31:25.679544 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:25.679516    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"
Nov 02 08:31:25.702715 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:25.702684    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13a7ae74-c294-4de1-b499-a2e0a7318cfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a7ae74-c294-4de1-b499-a2e0a7318cfc-plugins-config-volume\") pod \"13a7ae74-c294-4de1-b499-a2e0a7318cfc\" (UID: \"13a7ae74-c294-4de1-b499-a2e0a7318cfc\") "
Nov 02 08:31:25.702940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:25.702904    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a7ae74-c294-4de1-b499-a2e0a7318cfc-kube-api-access-26wnf\" (UniqueName: \"kubernetes.io/projected/13a7ae74-c294-4de1-b499-a2e0a7318cfc-kube-api-access-26wnf\") pod \"13a7ae74-c294-4de1-b499-a2e0a7318cfc\" (UID: \"13a7ae74-c294-4de1-b499-a2e0a7318cfc\") "
Nov 02 08:31:25.703094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:25.703072    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a7ae74-c294-4de1-b499-a2e0a7318cfc-plugins-config-volume" pod "13a7ae74-c294-4de1-b499-a2e0a7318cfc" (UID: "13a7ae74-c294-4de1-b499-a2e0a7318cfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:25.703343 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:25.703327    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a7ae74-c294-4de1-b499-a2e0a7318cfc-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:25.705363 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:25.705329    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a7ae74-c294-4de1-b499-a2e0a7318cfc-kube-api-access-26wnf" pod "13a7ae74-c294-4de1-b499-a2e0a7318cfc" (UID: "13a7ae74-c294-4de1-b499-a2e0a7318cfc"). InnerVolumeSpecName "kube-api-access-26wnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:25.804603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:25.804566    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26wnf\" (UniqueName: \"kubernetes.io/projected/13a7ae74-c294-4de1-b499-a2e0a7318cfc-kube-api-access-26wnf\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:26.371601 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:26.371560    4898 generic.go:356] "Generic (PLEG): container finished" podID="13a7ae74-c294-4de1-b499-a2e0a7318cfc" containerID="ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d" exitCode=0
Nov 02 08:31:26.372160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:26.371616    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"
Nov 02 08:31:26.372160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:26.371645    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj" event={"ID":"13a7ae74-c294-4de1-b499-a2e0a7318cfc","Type":"ContainerDied","Data":"ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d"}
Nov 02 08:31:26.372160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:26.371696    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj" event={"ID":"13a7ae74-c294-4de1-b499-a2e0a7318cfc","Type":"ContainerDied","Data":"70cc412417127bc73aa3d0dabab323710854590ee196e6eea69e3787632c475c"}
Nov 02 08:31:26.372160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:26.371716    4898 scope.go:122] "RemoveContainer" containerID="ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d"
Nov 02 08:31:26.388789 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:26.388768    4898 scope.go:122] "RemoveContainer" containerID="ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d"
Nov 02 08:31:26.389158 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:31:26.389133    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d\": container with ID starting with ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d not found: ID does not exist" containerID="ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d"
Nov 02 08:31:26.389295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:26.389165    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d"} err="failed to get container status \"ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d\": rpc error: code = NotFound desc = could not find container \"ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d\": container with ID starting with ba0010ee380fd44d8cef436199538e0169d04c8fb4d98fee136503334c8a396d not found: ID does not exist"
Nov 02 08:31:26.423782 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:26.423754    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"]
Nov 02 08:31:26.436026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:26.436001    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3575b9-7fd5656bb5-vldtj"]
Nov 02 08:31:27.467719 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:27.467678    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a7ae74-c294-4de1-b499-a2e0a7318cfc" path="/var/lib/kubelet/pods/13a7ae74-c294-4de1-b499-a2e0a7318cfc/volumes"
Nov 02 08:31:33.858166 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858123    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"]
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858798    4898 cpu_manager.go:414] "containerMap: removing container" podUID="045dedc0-929e-4caf-8349-11463c81fc48" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858821    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045dedc0-929e-4caf-8349-11463c81fc48" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858843    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerName="istio-proxy"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858851    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerName="istio-proxy"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858870    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858876    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858902    4898 cpu_manager.go:414] "containerMap: removing container" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858909    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b72892-6169-4ffa-bc4a-e64ad46688ae" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858918    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858925    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858960    4898 cpu_manager.go:414] "containerMap: removing container" podUID="eb7a2f08-f10d-4430-b0be-60892b129043" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858967    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7a2f08-f10d-4430-b0be-60892b129043" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858979    4898 cpu_manager.go:414] "containerMap: removing container" podUID="13a7ae74-c294-4de1-b499-a2e0a7318cfc" containerName="payload-processing"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.858986    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a7ae74-c294-4de1-b499-a2e0a7318cfc" containerName="payload-processing"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859002    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859008    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a259b571-ddfe-400f-a22b-7f16768a82f7" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859022    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859027    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859037    4898 cpu_manager.go:414] "containerMap: removing container" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859041    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb4d901-47b4-4247-b099-db9bd7144ce2" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859050    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859054    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859064    4898 cpu_manager.go:414] "containerMap: removing container" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859070    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859075    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859080    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859090    4898 cpu_manager.go:414] "containerMap: removing container" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859094    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6defcb68-a6cc-44f8-a5c2-fb40a2e8bb32" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859103    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859108    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c33009d-0656-4873-9be6-968f615c990d" containerName="istio-proxy"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859125    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d8398fcd-e412-4abe-b630-78573f4cf7fe" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859131    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8398fcd-e412-4abe-b630-78573f4cf7fe" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859282    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0539990-8490-41d3-8dba-77f7e8a6af14" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859293    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e09d271-5894-4f0d-9a38-dca2b8efb7bb" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859299    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3109b9e6-5dfe-4422-a630-b9c42c6bc1ab" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859304    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="045dedc0-929e-4caf-8349-11463c81fc48" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859311    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85b69ee-2f64-45f1-af4e-9f23b122f2c8" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859316    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f9a6a2-8c24-4edb-a91c-b2d906ddac6f" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859321    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abea0a3-c4f1-40df-af27-a75342569b3b" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859328    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70ee1e4-32ac-433e-80e5-4ab92fd76d0d" containerName="istio-proxy"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859335    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d7ab63-f8af-475d-a6e8-67e7d5587603" containerName="cleanup"
Nov 02 08:31:33.860131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.859344    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a7ae74-c294-4de1-b499-a2e0a7318cfc" containerName="payload-processing"
Nov 02 08:31:33.864654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.864628    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:33.893296 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.893260    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:31:33.893490 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.893445    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:31:33.900494 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.897384    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:31:33.900494 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.897512    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:31:33.900494 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.897713    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"]
Nov 02 08:31:33.909913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.909690    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"]
Nov 02 08:31:33.910106 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.909942    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy" containerID="cri-o://cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34" gracePeriod=30
Nov 02 08:31:33.954466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.954434    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb134db2-6gqsh\" (UID: \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:33.954630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.954559    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb134db2-6gqsh\" (UID: \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:33.954630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:33.954593    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6jt\" (UniqueName: \"kubernetes.io/projected/19b42405-696b-4aa8-b0e5-22c9556f9ddc-kube-api-access-6q6jt\") pod \"maas-api-revoke-sub-fb134db2-6gqsh\" (UID: \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:34.056921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:34.056850    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb134db2-6gqsh\" (UID: \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:34.057371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:34.057335    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb134db2-6gqsh\" (UID: \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:34.057604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:34.057574    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6jt\" (UniqueName: \"kubernetes.io/projected/19b42405-696b-4aa8-b0e5-22c9556f9ddc-kube-api-access-6q6jt\") pod \"maas-api-revoke-sub-fb134db2-6gqsh\" (UID: \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:34.070924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:34.070875    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb134db2-6gqsh\" (UID: \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:34.090649 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:34.090617    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb134db2-6gqsh\" (UID: \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:34.093817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:34.093787    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6jt\" (UniqueName: \"kubernetes.io/projected/19b42405-696b-4aa8-b0e5-22c9556f9ddc-kube-api-access-6q6jt\") pod \"maas-api-revoke-sub-fb134db2-6gqsh\" (UID: \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:34.189015 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:34.188978    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:34.500327 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:34.500287    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"]
Nov 02 08:31:34.501104 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:31:34.501074    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b42405_696b_4aa8_b0e5_22c9556f9ddc.slice/crio-019240afc9ea0482fa3d2c7d7a4c84697799195f930ec0f5fa01069372e019ea WatchSource:0}: Error finding container 019240afc9ea0482fa3d2c7d7a4c84697799195f930ec0f5fa01069372e019ea: Status 404 returned error can't find the container with id 019240afc9ea0482fa3d2c7d7a4c84697799195f930ec0f5fa01069372e019ea
Nov 02 08:31:34.820779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:34.820696    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"]
Nov 02 08:31:34.826286 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:34.826265    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:34.844238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:34.844207    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"]
Nov 02 08:31:35.003107 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.003069    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-150a71d3-bv2xs\" (UID: \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:35.003637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.003202    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-150a71d3-bv2xs\" (UID: \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:35.003637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.003360    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7fd9\" (UniqueName: \"kubernetes.io/projected/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-kube-api-access-t7fd9\") pod \"maas-api-revoke-sub-150a71d3-bv2xs\" (UID: \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:35.104340 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.104263    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-150a71d3-bv2xs\" (UID: \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:35.104510 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.104404    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7fd9\" (UniqueName: \"kubernetes.io/projected/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-kube-api-access-t7fd9\") pod \"maas-api-revoke-sub-150a71d3-bv2xs\" (UID: \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:35.104510 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.104474    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-150a71d3-bv2xs\" (UID: \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:35.105168 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.105137    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-150a71d3-bv2xs\" (UID: \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:35.106822 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.106794    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-150a71d3-bv2xs\" (UID: \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:35.127415 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.127390    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7fd9\" (UniqueName: \"kubernetes.io/projected/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-kube-api-access-t7fd9\") pod \"maas-api-revoke-sub-150a71d3-bv2xs\" (UID: \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:35.141305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.141283    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:35.455981 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:31:35.455935    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67d9e0d_ff4d_4ecc_bbd9_e65b064c0d47.slice/crio-a85fd51a7c774d4c9871bc74e8524356cee478ef0655c12452c2246685e3b0c2 WatchSource:0}: Error finding container a85fd51a7c774d4c9871bc74e8524356cee478ef0655c12452c2246685e3b0c2: Status 404 returned error can't find the container with id a85fd51a7c774d4c9871bc74e8524356cee478ef0655c12452c2246685e3b0c2
Nov 02 08:31:35.468207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.468171    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh" event={"ID":"19b42405-696b-4aa8-b0e5-22c9556f9ddc","Type":"ContainerStarted","Data":"c808b2f49791e9cfdbb343998d7d20a7e077486569d4a11bef50c79baecb7cc6"}
Nov 02 08:31:35.468316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.468234    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"]
Nov 02 08:31:35.468316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.468269    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh" event={"ID":"19b42405-696b-4aa8-b0e5-22c9556f9ddc","Type":"ContainerStarted","Data":"019240afc9ea0482fa3d2c7d7a4c84697799195f930ec0f5fa01069372e019ea"}
Nov 02 08:31:35.469881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.469856    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs" event={"ID":"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47","Type":"ContainerStarted","Data":"a85fd51a7c774d4c9871bc74e8524356cee478ef0655c12452c2246685e3b0c2"}
Nov 02 08:31:35.509437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:35.509383    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh" podStartSLOduration=2.509364107 podStartE2EDuration="2.509364107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:35.507948609 +0000 UTC m=+4046.198275574" watchObservedRunningTime="2026-11-02 08:31:35.509364107 +0000 UTC m=+4046.199691078"
Nov 02 08:31:36.480566 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:36.480525    4898 generic.go:356] "Generic (PLEG): container finished" podID="19b42405-696b-4aa8-b0e5-22c9556f9ddc" containerID="c808b2f49791e9cfdbb343998d7d20a7e077486569d4a11bef50c79baecb7cc6" exitCode=0
Nov 02 08:31:36.481090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:36.480611    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh" event={"ID":"19b42405-696b-4aa8-b0e5-22c9556f9ddc","Type":"ContainerDied","Data":"c808b2f49791e9cfdbb343998d7d20a7e077486569d4a11bef50c79baecb7cc6"}
Nov 02 08:31:36.482512 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:36.482486    4898 generic.go:356] "Generic (PLEG): container finished" podID="c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47" containerID="c3b92c1956a996d8c15dc222ed09569473c2968ba6883515370755da4c04cfa2" exitCode=0
Nov 02 08:31:36.482639 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:36.482567    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs" event={"ID":"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47","Type":"ContainerDied","Data":"c3b92c1956a996d8c15dc222ed09569473c2968ba6883515370755da4c04cfa2"}
Nov 02 08:31:37.047190 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.047148    4898 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:31:37.047360 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.047215    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:31:37.706759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.706721    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:37.710238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.710216    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:37.843127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.843025    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b42405-696b-4aa8-b0e5-22c9556f9ddc-kube-api-access-6q6jt\" (UniqueName: \"kubernetes.io/projected/19b42405-696b-4aa8-b0e5-22c9556f9ddc-kube-api-access-6q6jt\") pod \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\" (UID: \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\") "
Nov 02 08:31:37.843127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.843091    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-xks-ca\") pod \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\" (UID: \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\") "
Nov 02 08:31:37.843365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.843248    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-xks-ca\") pod \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\" (UID: \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\") "
Nov 02 08:31:37.843431 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.843364    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-openshift-ca\") pod \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\" (UID: \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\") "
Nov 02 08:31:37.843431 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.843406    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-kube-api-access-t7fd9\" (UniqueName: \"kubernetes.io/projected/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-kube-api-access-t7fd9\") pod \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\" (UID: \"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47\") "
Nov 02 08:31:37.843549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.843525    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-openshift-ca\") pod \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\" (UID: \"19b42405-696b-4aa8-b0e5-22c9556f9ddc\") "
Nov 02 08:31:37.844216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.843760    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-openshift-ca" pod "c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47" (UID: "c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:37.844216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.844135    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-openshift-ca" pod "19b42405-696b-4aa8-b0e5-22c9556f9ddc" (UID: "19b42405-696b-4aa8-b0e5-22c9556f9ddc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:37.844216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.844203    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:37.845624 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.845573    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-xks-ca" pod "19b42405-696b-4aa8-b0e5-22c9556f9ddc" (UID: "19b42405-696b-4aa8-b0e5-22c9556f9ddc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:31:37.845734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.845650    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-xks-ca" pod "c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47" (UID: "c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:31:37.845734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.845688    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-kube-api-access-t7fd9" pod "c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47" (UID: "c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47"). InnerVolumeSpecName "kube-api-access-t7fd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:37.845823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.845798    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b42405-696b-4aa8-b0e5-22c9556f9ddc-kube-api-access-6q6jt" pod "19b42405-696b-4aa8-b0e5-22c9556f9ddc" (UID: "19b42405-696b-4aa8-b0e5-22c9556f9ddc"). InnerVolumeSpecName "kube-api-access-6q6jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:37.945826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.945782    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:37.945826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.945824    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7fd9\" (UniqueName: \"kubernetes.io/projected/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47-kube-api-access-t7fd9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:37.945826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.945838    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:37.946110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.945852    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6jt\" (UniqueName: \"kubernetes.io/projected/19b42405-696b-4aa8-b0e5-22c9556f9ddc-kube-api-access-6q6jt\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:37.946110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:37.945866    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b42405-696b-4aa8-b0e5-22c9556f9ddc-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:38.503253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:38.503223    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"
Nov 02 08:31:38.503253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:38.503235    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh" event={"ID":"19b42405-696b-4aa8-b0e5-22c9556f9ddc","Type":"ContainerDied","Data":"019240afc9ea0482fa3d2c7d7a4c84697799195f930ec0f5fa01069372e019ea"}
Nov 02 08:31:38.503498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:38.503285    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="019240afc9ea0482fa3d2c7d7a4c84697799195f930ec0f5fa01069372e019ea"
Nov 02 08:31:38.504828 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:38.504809    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"
Nov 02 08:31:38.505003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:38.504843    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs" event={"ID":"c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47","Type":"ContainerDied","Data":"a85fd51a7c774d4c9871bc74e8524356cee478ef0655c12452c2246685e3b0c2"}
Nov 02 08:31:38.505003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:38.504916    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a85fd51a7c774d4c9871bc74e8524356cee478ef0655c12452c2246685e3b0c2"
Nov 02 08:31:39.215493 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.215469    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 08:31:39.359880 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.359836    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"]
Nov 02 08:31:39.363914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.363857    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-credential-socket\") pod \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") "
Nov 02 08:31:39.364087 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.363929    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-kube-api-access-s92vn\" (UniqueName: \"kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-kube-api-access-s92vn\") pod \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") "
Nov 02 08:31:39.364165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.364088    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-podinfo\") pod \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") "
Nov 02 08:31:39.364165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.364142    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-certs\") pod \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") "
Nov 02 08:31:39.364298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.364198    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-credential-socket" pod "529b9b31-ca08-40e9-815b-5f5afee7dd1d" (UID: "529b9b31-ca08-40e9-815b-5f5afee7dd1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:39.364298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.364204    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-socket\") pod \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") "
Nov 02 08:31:39.364408 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.364326    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istiod-ca-cert\") pod \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") "
Nov 02 08:31:39.364483 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.364411    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-data\") pod \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") "
Nov 02 08:31:39.364546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.364488    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-envoy\") pod \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") "
Nov 02 08:31:39.364546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.364528    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-token\" (UniqueName: \"kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-token\") pod \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\" (UID: \"529b9b31-ca08-40e9-815b-5f5afee7dd1d\") "
Nov 02 08:31:39.365336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.364908    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-socket" pod "529b9b31-ca08-40e9-815b-5f5afee7dd1d" (UID: "529b9b31-ca08-40e9-815b-5f5afee7dd1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:39.365537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.365306    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:39.365629 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.365552    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:39.365629 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.365309    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istiod-ca-cert" pod "529b9b31-ca08-40e9-815b-5f5afee7dd1d" (UID: "529b9b31-ca08-40e9-815b-5f5afee7dd1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:39.365802 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.365634    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-data" pod "529b9b31-ca08-40e9-815b-5f5afee7dd1d" (UID: "529b9b31-ca08-40e9-815b-5f5afee7dd1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:39.365802 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.365700    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-certs" pod "529b9b31-ca08-40e9-815b-5f5afee7dd1d" (UID: "529b9b31-ca08-40e9-815b-5f5afee7dd1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:39.368432 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.368396    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-envoy" pod "529b9b31-ca08-40e9-815b-5f5afee7dd1d" (UID: "529b9b31-ca08-40e9-815b-5f5afee7dd1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:39.368700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.368674    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-token" pod "529b9b31-ca08-40e9-815b-5f5afee7dd1d" (UID: "529b9b31-ca08-40e9-815b-5f5afee7dd1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:39.368857 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.368792    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-kube-api-access-s92vn" pod "529b9b31-ca08-40e9-815b-5f5afee7dd1d" (UID: "529b9b31-ca08-40e9-815b-5f5afee7dd1d"). InnerVolumeSpecName "kube-api-access-s92vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:39.369181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.369158    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"]
Nov 02 08:31:39.369362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.369341    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-podinfo" pod "529b9b31-ca08-40e9-815b-5f5afee7dd1d" (UID: "529b9b31-ca08-40e9-815b-5f5afee7dd1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:31:39.379053 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.379027    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"]
Nov 02 08:31:39.467047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.466965    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s92vn\" (UniqueName: \"kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-kube-api-access-s92vn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:39.467047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.466995    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:39.467047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.467004    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:39.467047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.467014    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:39.467047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.467027    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:39.467047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.467037    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:39.467047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.467045    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529b9b31-ca08-40e9-815b-5f5afee7dd1d-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:39.516382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.516340    4898 generic.go:356] "Generic (PLEG): container finished" podID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerID="cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34" exitCode=0
Nov 02 08:31:39.516564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.516402    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"
Nov 02 08:31:39.516564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.516425    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" event={"ID":"529b9b31-ca08-40e9-815b-5f5afee7dd1d","Type":"ContainerDied","Data":"cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34"}
Nov 02 08:31:39.516564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.516490    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25" event={"ID":"529b9b31-ca08-40e9-815b-5f5afee7dd1d","Type":"ContainerDied","Data":"e2ceaf19bb912160a38ea297562fbaf3dabb45b96ae19bc034c991576d423345"}
Nov 02 08:31:39.516564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.516520    4898 scope.go:122] "RemoveContainer" containerID="cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34"
Nov 02 08:31:39.528389 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.528371    4898 scope.go:122] "RemoveContainer" containerID="a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef"
Nov 02 08:31:39.540628 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.540606    4898 scope.go:122] "RemoveContainer" containerID="cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34"
Nov 02 08:31:39.540976 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:31:39.540879    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34\": container with ID starting with cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34 not found: ID does not exist" containerID="cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34"
Nov 02 08:31:39.541025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.540985    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34"} err="failed to get container status \"cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34\": rpc error: code = NotFound desc = could not find container \"cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34\": container with ID starting with cacd4d5e2e6d648d26787515080589d80a03b7f50c6c4a920807f99d0f978e34 not found: ID does not exist"
Nov 02 08:31:39.541025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.541003    4898 scope.go:122] "RemoveContainer" containerID="a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef"
Nov 02 08:31:39.541363 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:31:39.541341    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef\": container with ID starting with a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef not found: ID does not exist" containerID="a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef"
Nov 02 08:31:39.541437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.541372    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef"} err="failed to get container status \"a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef\": rpc error: code = NotFound desc = could not find container \"a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef\": container with ID starting with a7f573ba98bfa388e2fbbe24fc90a5410cb4a83fcb0db3f396924fc45ffe71ef not found: ID does not exist"
Nov 02 08:31:39.571034 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.570998    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"]
Nov 02 08:31:39.574786 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:39.574764    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3575b9-openshift-default-6d68fdfcb7-dpd25"]
Nov 02 08:31:41.467509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:41.467470    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" path="/var/lib/kubelet/pods/529b9b31-ca08-40e9-815b-5f5afee7dd1d/volumes"
Nov 02 08:31:44.381024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.378225    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"]
Nov 02 08:31:44.381024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.379409    4898 cpu_manager.go:414] "containerMap: removing container" podUID="19b42405-696b-4aa8-b0e5-22c9556f9ddc" containerName="revoke-keys"
Nov 02 08:31:44.381024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.379428    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b42405-696b-4aa8-b0e5-22c9556f9ddc" containerName="revoke-keys"
Nov 02 08:31:44.381024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.379464    4898 cpu_manager.go:414] "containerMap: removing container" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy"
Nov 02 08:31:44.381024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.379470    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy"
Nov 02 08:31:44.381024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.379521    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47" containerName="revoke-keys"
Nov 02 08:31:44.381024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.379530    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47" containerName="revoke-keys"
Nov 02 08:31:44.381024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.379551    4898 cpu_manager.go:414] "containerMap: removing container" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy"
Nov 02 08:31:44.381024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.379556    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy"
Nov 02 08:31:44.381024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.379752    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47" containerName="revoke-keys"
Nov 02 08:31:44.381024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.379775    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy"
Nov 02 08:31:44.381024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.379788    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b42405-696b-4aa8-b0e5-22c9556f9ddc" containerName="revoke-keys"
Nov 02 08:31:44.389440 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.389415    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"]
Nov 02 08:31:44.389609 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.389537    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:44.410626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.410597    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:31:44.421620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.421597    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:31:44.421620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.421609    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:31:44.421785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.421637    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:31:44.444603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.444574    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76fb09e5-4phpf\" (UID: \"4538bddc-7e27-4ac5-a370-29cf46d6669a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:44.444753 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.444637    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8ms\" (UniqueName: \"kubernetes.io/projected/4538bddc-7e27-4ac5-a370-29cf46d6669a-kube-api-access-bx8ms\") pod \"maas-api-revoke-sub-76fb09e5-4phpf\" (UID: \"4538bddc-7e27-4ac5-a370-29cf46d6669a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:44.444753 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.444671    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76fb09e5-4phpf\" (UID: \"4538bddc-7e27-4ac5-a370-29cf46d6669a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:44.545806 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.545772    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76fb09e5-4phpf\" (UID: \"4538bddc-7e27-4ac5-a370-29cf46d6669a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:44.546213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.546183    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8ms\" (UniqueName: \"kubernetes.io/projected/4538bddc-7e27-4ac5-a370-29cf46d6669a-kube-api-access-bx8ms\") pod \"maas-api-revoke-sub-76fb09e5-4phpf\" (UID: \"4538bddc-7e27-4ac5-a370-29cf46d6669a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:44.546418 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.546400    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76fb09e5-4phpf\" (UID: \"4538bddc-7e27-4ac5-a370-29cf46d6669a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:44.546721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.546692    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76fb09e5-4phpf\" (UID: \"4538bddc-7e27-4ac5-a370-29cf46d6669a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:44.548972 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.548944    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76fb09e5-4phpf\" (UID: \"4538bddc-7e27-4ac5-a370-29cf46d6669a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:44.570043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.570014    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8ms\" (UniqueName: \"kubernetes.io/projected/4538bddc-7e27-4ac5-a370-29cf46d6669a-kube-api-access-bx8ms\") pod \"maas-api-revoke-sub-76fb09e5-4phpf\" (UID: \"4538bddc-7e27-4ac5-a370-29cf46d6669a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:44.714320 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:44.714286    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:44.994777 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:31:44.994673    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:31:45.035851 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:45.035812    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"]
Nov 02 08:31:45.036715 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:31:45.036625    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4538bddc_7e27_4ac5_a370_29cf46d6669a.slice/crio-74790a3e76299f616fbebeef4be2eba2fefb57800fead3948d805b1c7cdb1307 WatchSource:0}: Error finding container 74790a3e76299f616fbebeef4be2eba2fefb57800fead3948d805b1c7cdb1307: Status 404 returned error can't find the container with id 74790a3e76299f616fbebeef4be2eba2fefb57800fead3948d805b1c7cdb1307
Nov 02 08:31:45.576908 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:45.576800    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf" event={"ID":"4538bddc-7e27-4ac5-a370-29cf46d6669a","Type":"ContainerStarted","Data":"c10d8ea9f9fe7ce6f681878658626b2c5b5359bc96411f65476142fd026785a1"}
Nov 02 08:31:45.576908 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:45.576868    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf" event={"ID":"4538bddc-7e27-4ac5-a370-29cf46d6669a","Type":"ContainerStarted","Data":"74790a3e76299f616fbebeef4be2eba2fefb57800fead3948d805b1c7cdb1307"}
Nov 02 08:31:45.631711 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:45.631643    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf" podStartSLOduration=1.631623547 podStartE2EDuration="1.631623547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:45.627176008 +0000 UTC m=+4056.317502974" watchObservedRunningTime="2026-11-02 08:31:45.631623547 +0000 UTC m=+4056.321950513"
Nov 02 08:31:46.588557 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:46.588519    4898 generic.go:356] "Generic (PLEG): container finished" podID="4538bddc-7e27-4ac5-a370-29cf46d6669a" containerID="c10d8ea9f9fe7ce6f681878658626b2c5b5359bc96411f65476142fd026785a1" exitCode=0
Nov 02 08:31:46.589026 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:46.588608    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf" event={"ID":"4538bddc-7e27-4ac5-a370-29cf46d6669a","Type":"ContainerDied","Data":"c10d8ea9f9fe7ce6f681878658626b2c5b5359bc96411f65476142fd026785a1"}
Nov 02 08:31:47.553260 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:47.553228    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"]
Nov 02 08:31:47.587291 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:47.587258    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"]
Nov 02 08:31:47.832875 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:47.832771    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:47.918995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:47.918961    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4538bddc-7e27-4ac5-a370-29cf46d6669a-kube-api-access-bx8ms\" (UniqueName: \"kubernetes.io/projected/4538bddc-7e27-4ac5-a370-29cf46d6669a-kube-api-access-bx8ms\") pod \"4538bddc-7e27-4ac5-a370-29cf46d6669a\" (UID: \"4538bddc-7e27-4ac5-a370-29cf46d6669a\") "
Nov 02 08:31:47.919181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:47.919135    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-xks-ca\") pod \"4538bddc-7e27-4ac5-a370-29cf46d6669a\" (UID: \"4538bddc-7e27-4ac5-a370-29cf46d6669a\") "
Nov 02 08:31:47.919269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:47.919180    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-openshift-ca\") pod \"4538bddc-7e27-4ac5-a370-29cf46d6669a\" (UID: \"4538bddc-7e27-4ac5-a370-29cf46d6669a\") "
Nov 02 08:31:47.920037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:47.919997    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-openshift-ca" pod "4538bddc-7e27-4ac5-a370-29cf46d6669a" (UID: "4538bddc-7e27-4ac5-a370-29cf46d6669a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:47.920302 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:47.920282    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:47.921927 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:47.921869    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4538bddc-7e27-4ac5-a370-29cf46d6669a-kube-api-access-bx8ms" pod "4538bddc-7e27-4ac5-a370-29cf46d6669a" (UID: "4538bddc-7e27-4ac5-a370-29cf46d6669a"). InnerVolumeSpecName "kube-api-access-bx8ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:47.922254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:47.922229    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-xks-ca" pod "4538bddc-7e27-4ac5-a370-29cf46d6669a" (UID: "4538bddc-7e27-4ac5-a370-29cf46d6669a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:31:48.022071 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:48.022028    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx8ms\" (UniqueName: \"kubernetes.io/projected/4538bddc-7e27-4ac5-a370-29cf46d6669a-kube-api-access-bx8ms\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:48.022071 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:48.022064    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4538bddc-7e27-4ac5-a370-29cf46d6669a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:48.607523 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:48.607497    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"
Nov 02 08:31:48.607716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:48.607496    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf" event={"ID":"4538bddc-7e27-4ac5-a370-29cf46d6669a","Type":"ContainerDied","Data":"74790a3e76299f616fbebeef4be2eba2fefb57800fead3948d805b1c7cdb1307"}
Nov 02 08:31:48.607716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:48.607621    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74790a3e76299f616fbebeef4be2eba2fefb57800fead3948d805b1c7cdb1307"
Nov 02 08:31:49.305542 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.305498    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"]
Nov 02 08:31:49.306371 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.306348    4898 cpu_manager.go:414] "containerMap: removing container" podUID="4538bddc-7e27-4ac5-a370-29cf46d6669a" containerName="revoke-keys"
Nov 02 08:31:49.306482 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.306372    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4538bddc-7e27-4ac5-a370-29cf46d6669a" containerName="revoke-keys"
Nov 02 08:31:49.306565 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.306550    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="529b9b31-ca08-40e9-815b-5f5afee7dd1d" containerName="istio-proxy"
Nov 02 08:31:49.306624 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.306586    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4538bddc-7e27-4ac5-a370-29cf46d6669a" containerName="revoke-keys"
Nov 02 08:31:49.310223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.310194    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:49.328029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.328002    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"]
Nov 02 08:31:49.336249 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.336227    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:31:49.336402 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.336295    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:31:49.336502 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.336364    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:31:49.337862 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.337839    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:31:49.443407 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.443374    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mxqw\" (UniqueName: \"kubernetes.io/projected/acdd4d3a-3be4-4524-8622-217cf531d6cf-kube-api-access-7mxqw\") pod \"maas-api-revoke-sub-82a36a4e-jsqgq\" (UID: \"acdd4d3a-3be4-4524-8622-217cf531d6cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:49.443621 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.443524    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a36a4e-jsqgq\" (UID: \"acdd4d3a-3be4-4524-8622-217cf531d6cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:49.443621 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.443559    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a36a4e-jsqgq\" (UID: \"acdd4d3a-3be4-4524-8622-217cf531d6cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:49.545285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.545251    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a36a4e-jsqgq\" (UID: \"acdd4d3a-3be4-4524-8622-217cf531d6cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:49.545456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.545298    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a36a4e-jsqgq\" (UID: \"acdd4d3a-3be4-4524-8622-217cf531d6cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:49.545556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.545497    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mxqw\" (UniqueName: \"kubernetes.io/projected/acdd4d3a-3be4-4524-8622-217cf531d6cf-kube-api-access-7mxqw\") pod \"maas-api-revoke-sub-82a36a4e-jsqgq\" (UID: \"acdd4d3a-3be4-4524-8622-217cf531d6cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:49.546022 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.546000    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a36a4e-jsqgq\" (UID: \"acdd4d3a-3be4-4524-8622-217cf531d6cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:49.547866 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.547843    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a36a4e-jsqgq\" (UID: \"acdd4d3a-3be4-4524-8622-217cf531d6cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:49.568788 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.568732    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mxqw\" (UniqueName: \"kubernetes.io/projected/acdd4d3a-3be4-4524-8622-217cf531d6cf-kube-api-access-7mxqw\") pod \"maas-api-revoke-sub-82a36a4e-jsqgq\" (UID: \"acdd4d3a-3be4-4524-8622-217cf531d6cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:49.625216 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.625190    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:49.939457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:49.939429    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"]
Nov 02 08:31:49.939739 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:31:49.939719    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacdd4d3a_3be4_4524_8622_217cf531d6cf.slice/crio-b98f6a198628914b96409336ad0d3dc619894ee51b4466586910b789a7cb6f19 WatchSource:0}: Error finding container b98f6a198628914b96409336ad0d3dc619894ee51b4466586910b789a7cb6f19: Status 404 returned error can't find the container with id b98f6a198628914b96409336ad0d3dc619894ee51b4466586910b789a7cb6f19
Nov 02 08:31:50.631944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:50.631820    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq" event={"ID":"acdd4d3a-3be4-4524-8622-217cf531d6cf","Type":"ContainerStarted","Data":"5d63409a5d50118fa665eba0f3dfbbcbb91b1c4ea68433becad052996d9a8680"}
Nov 02 08:31:50.631944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:50.631938    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq" event={"ID":"acdd4d3a-3be4-4524-8622-217cf531d6cf","Type":"ContainerStarted","Data":"b98f6a198628914b96409336ad0d3dc619894ee51b4466586910b789a7cb6f19"}
Nov 02 08:31:50.678820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:50.678757    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq" podStartSLOduration=1.678740387 podStartE2EDuration="1.678740387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:31:50.671537573 +0000 UTC m=+4061.361864540" watchObservedRunningTime="2026-11-02 08:31:50.678740387 +0000 UTC m=+4061.369067352"
Nov 02 08:31:51.640597 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:51.640562    4898 generic.go:356] "Generic (PLEG): container finished" podID="acdd4d3a-3be4-4524-8622-217cf531d6cf" containerID="5d63409a5d50118fa665eba0f3dfbbcbb91b1c4ea68433becad052996d9a8680" exitCode=0
Nov 02 08:31:51.641036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:51.640650    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq" event={"ID":"acdd4d3a-3be4-4524-8622-217cf531d6cf","Type":"ContainerDied","Data":"5d63409a5d50118fa665eba0f3dfbbcbb91b1c4ea68433becad052996d9a8680"}
Nov 02 08:31:52.554063 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:52.554023    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5" podUID="c9606134-fab6-48c1-9543-b20f0cf28967" containerName="payload-processing" containerID="cri-o://e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24" gracePeriod=25
Nov 02 08:31:52.587958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:52.587863    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4" podUID="ea1f4f74-4502-49dd-a54a-893b9264a0cf" containerName="payload-pre-processing" containerID="cri-o://1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8" gracePeriod=25
Nov 02 08:31:52.819462 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:52.819402    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:52.900852 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:52.900813    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-openshift-ca\") pod \"acdd4d3a-3be4-4524-8622-217cf531d6cf\" (UID: \"acdd4d3a-3be4-4524-8622-217cf531d6cf\") "
Nov 02 08:31:52.901296 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:52.901265    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-openshift-ca" pod "acdd4d3a-3be4-4524-8622-217cf531d6cf" (UID: "acdd4d3a-3be4-4524-8622-217cf531d6cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:52.901393 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:52.901331    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acdd4d3a-3be4-4524-8622-217cf531d6cf-kube-api-access-7mxqw\" (UniqueName: \"kubernetes.io/projected/acdd4d3a-3be4-4524-8622-217cf531d6cf-kube-api-access-7mxqw\") pod \"acdd4d3a-3be4-4524-8622-217cf531d6cf\" (UID: \"acdd4d3a-3be4-4524-8622-217cf531d6cf\") "
Nov 02 08:31:52.901586 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:52.901556    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-xks-ca\") pod \"acdd4d3a-3be4-4524-8622-217cf531d6cf\" (UID: \"acdd4d3a-3be4-4524-8622-217cf531d6cf\") "
Nov 02 08:31:52.902218 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:52.902195    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:52.905108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:52.905080    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-xks-ca" pod "acdd4d3a-3be4-4524-8622-217cf531d6cf" (UID: "acdd4d3a-3be4-4524-8622-217cf531d6cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:31:52.905710 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:52.905681    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdd4d3a-3be4-4524-8622-217cf531d6cf-kube-api-access-7mxqw" pod "acdd4d3a-3be4-4524-8622-217cf531d6cf" (UID: "acdd4d3a-3be4-4524-8622-217cf531d6cf"). InnerVolumeSpecName "kube-api-access-7mxqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:53.003991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:53.003950    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mxqw\" (UniqueName: \"kubernetes.io/projected/acdd4d3a-3be4-4524-8622-217cf531d6cf-kube-api-access-7mxqw\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:53.003991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:53.003981    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acdd4d3a-3be4-4524-8622-217cf531d6cf-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:53.663796 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:53.663763    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"
Nov 02 08:31:53.664389 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:53.663763    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq" event={"ID":"acdd4d3a-3be4-4524-8622-217cf531d6cf","Type":"ContainerDied","Data":"b98f6a198628914b96409336ad0d3dc619894ee51b4466586910b789a7cb6f19"}
Nov 02 08:31:53.664480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:53.664420    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b98f6a198628914b96409336ad0d3dc619894ee51b4466586910b789a7cb6f19"
Nov 02 08:31:54.408912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.406532    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" containerID="cri-o://429e49c5d6d57d3297caf95da943cc459a1018f4bba61946ce759d109a49a468" gracePeriod=45
Nov 02 08:31:54.408912 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.408252    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4" podUID="ea1f4f74-4502-49dd-a54a-893b9264a0cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:31:54.415292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.415175    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5" podUID="c9606134-fab6-48c1-9543-b20f0cf28967" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:31:54.415861 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.415834    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" containerID="cri-o://c6e808c347ec39d11c99a28a11c1f6b1a2fc93cfb1c3a01778e4ef253a5e12b2" gracePeriod=45
Nov 02 08:31:54.416132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.415834    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" containerID="cri-o://6333e57018fb396231aa307a5f430c9f990f5a44109c5cc34c1eb74bf4509de8" gracePeriod=45
Nov 02 08:31:54.487138 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.487092    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused"
Nov 02 08:31:54.710261 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.709875    4898 generic.go:356] "Generic (PLEG): container finished" podID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerID="429e49c5d6d57d3297caf95da943cc459a1018f4bba61946ce759d109a49a468" exitCode=0
Nov 02 08:31:54.710261 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.709980    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" event={"ID":"11ddd010-79e1-4281-b3ba-a5c0f273fde1","Type":"ContainerDied","Data":"429e49c5d6d57d3297caf95da943cc459a1018f4bba61946ce759d109a49a468"}
Nov 02 08:31:54.711997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.711564    4898 generic.go:356] "Generic (PLEG): container finished" podID="698187ec-6d8a-4096-abef-c51a70654104" containerID="6333e57018fb396231aa307a5f430c9f990f5a44109c5cc34c1eb74bf4509de8" exitCode=0
Nov 02 08:31:54.711997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.711759    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" event={"ID":"698187ec-6d8a-4096-abef-c51a70654104","Type":"ContainerDied","Data":"6333e57018fb396231aa307a5f430c9f990f5a44109c5cc34c1eb74bf4509de8"}
Nov 02 08:31:54.713404 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.713382    4898 generic.go:356] "Generic (PLEG): container finished" podID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerID="c6e808c347ec39d11c99a28a11c1f6b1a2fc93cfb1c3a01778e4ef253a5e12b2" exitCode=0
Nov 02 08:31:54.713512 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.713454    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" event={"ID":"34cee5d4-7fa9-4857-9916-49ab2a89f8e0","Type":"ContainerDied","Data":"c6e808c347ec39d11c99a28a11c1f6b1a2fc93cfb1c3a01778e4ef253a5e12b2"}
Nov 02 08:31:54.772337 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.771315    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 08:31:54.777021 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.776992    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 08:31:54.783076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.781062    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 08:31:54.947708 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.947674    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-model-cache\") pod \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") "
Nov 02 08:31:54.947878 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.947714    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tmp-dir\") pod \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") "
Nov 02 08:31:54.947878 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.947746    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tmp-dir\") pod \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") "
Nov 02 08:31:54.947878 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.947804    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-dshm\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-dshm\") pod \"698187ec-6d8a-4096-abef-c51a70654104\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") "
Nov 02 08:31:54.947878 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.947848    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-model-cache\") pod \"698187ec-6d8a-4096-abef-c51a70654104\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") "
Nov 02 08:31:54.948162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.947921    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-model-cache\") pod \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") "
Nov 02 08:31:54.948162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.947953    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698187ec-6d8a-4096-abef-c51a70654104-tls-certs\" (UniqueName: \"kubernetes.io/secret/698187ec-6d8a-4096-abef-c51a70654104-tls-certs\") pod \"698187ec-6d8a-4096-abef-c51a70654104\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") "
Nov 02 08:31:54.948162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.947978    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-tmp-dir\") pod \"698187ec-6d8a-4096-abef-c51a70654104\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") "
Nov 02 08:31:54.948162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948033    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-dshm\") pod \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") "
Nov 02 08:31:54.948162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948081    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tmp-dir" pod "11ddd010-79e1-4281-b3ba-a5c0f273fde1" (UID: "11ddd010-79e1-4281-b3ba-a5c0f273fde1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.948162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948105    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-home\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-home\") pod \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") "
Nov 02 08:31:54.948504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948204    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-kube-api-access-rthnm\" (UniqueName: \"kubernetes.io/projected/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-kube-api-access-rthnm\") pod \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") "
Nov 02 08:31:54.948504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948256    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ddd010-79e1-4281-b3ba-a5c0f273fde1-kube-api-access-vtz9h\" (UniqueName: \"kubernetes.io/projected/11ddd010-79e1-4281-b3ba-a5c0f273fde1-kube-api-access-vtz9h\") pod \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") "
Nov 02 08:31:54.948504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948317    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tls-certs\" (UniqueName: \"kubernetes.io/secret/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tls-certs\") pod \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\" (UID: \"11ddd010-79e1-4281-b3ba-a5c0f273fde1\") "
Nov 02 08:31:54.948504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948359    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-dshm\") pod \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") "
Nov 02 08:31:54.948504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948388    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-home\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-home\") pod \"698187ec-6d8a-4096-abef-c51a70654104\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") "
Nov 02 08:31:54.948504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948456    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tls-certs\") pod \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") "
Nov 02 08:31:54.948852 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948528    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-home\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-home\") pod \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\" (UID: \"34cee5d4-7fa9-4857-9916-49ab2a89f8e0\") "
Nov 02 08:31:54.948852 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948564    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698187ec-6d8a-4096-abef-c51a70654104-kube-api-access-5c5nj\" (UniqueName: \"kubernetes.io/projected/698187ec-6d8a-4096-abef-c51a70654104-kube-api-access-5c5nj\") pod \"698187ec-6d8a-4096-abef-c51a70654104\" (UID: \"698187ec-6d8a-4096-abef-c51a70654104\") "
Nov 02 08:31:54.948852 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948576    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-home" pod "11ddd010-79e1-4281-b3ba-a5c0f273fde1" (UID: "11ddd010-79e1-4281-b3ba-a5c0f273fde1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.948852 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948587    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tmp-dir" pod "34cee5d4-7fa9-4857-9916-49ab2a89f8e0" (UID: "34cee5d4-7fa9-4857-9916-49ab2a89f8e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.949115 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948971    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-model-cache" pod "698187ec-6d8a-4096-abef-c51a70654104" (UID: "698187ec-6d8a-4096-abef-c51a70654104"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.949398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.949370    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:54.949495 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.949403    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:54.949495 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.949417    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:54.949495 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.949428    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:54.949720 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.948517    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-tmp-dir" pod "698187ec-6d8a-4096-abef-c51a70654104" (UID: "698187ec-6d8a-4096-abef-c51a70654104"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.950305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.950262    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-model-cache" pod "34cee5d4-7fa9-4857-9916-49ab2a89f8e0" (UID: "34cee5d4-7fa9-4857-9916-49ab2a89f8e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.952271 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.952242    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-home" pod "698187ec-6d8a-4096-abef-c51a70654104" (UID: "698187ec-6d8a-4096-abef-c51a70654104"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.952952 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.952924    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-kube-api-access-rthnm" pod "34cee5d4-7fa9-4857-9916-49ab2a89f8e0" (UID: "34cee5d4-7fa9-4857-9916-49ab2a89f8e0"). InnerVolumeSpecName "kube-api-access-rthnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:54.953176 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.953151    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-model-cache" pod "11ddd010-79e1-4281-b3ba-a5c0f273fde1" (UID: "11ddd010-79e1-4281-b3ba-a5c0f273fde1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.953176 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.953165    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-home" pod "34cee5d4-7fa9-4857-9916-49ab2a89f8e0" (UID: "34cee5d4-7fa9-4857-9916-49ab2a89f8e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.953534 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.953500    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-dshm" pod "11ddd010-79e1-4281-b3ba-a5c0f273fde1" (UID: "11ddd010-79e1-4281-b3ba-a5c0f273fde1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.953716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.953665    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-dshm" pod "698187ec-6d8a-4096-abef-c51a70654104" (UID: "698187ec-6d8a-4096-abef-c51a70654104"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.953716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.953683    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698187ec-6d8a-4096-abef-c51a70654104-tls-certs" pod "698187ec-6d8a-4096-abef-c51a70654104" (UID: "698187ec-6d8a-4096-abef-c51a70654104"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:31:54.953716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.953692    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-dshm" pod "34cee5d4-7fa9-4857-9916-49ab2a89f8e0" (UID: "34cee5d4-7fa9-4857-9916-49ab2a89f8e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:31:54.954052 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.954029    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tls-certs" pod "11ddd010-79e1-4281-b3ba-a5c0f273fde1" (UID: "11ddd010-79e1-4281-b3ba-a5c0f273fde1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:31:54.955185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.955138    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698187ec-6d8a-4096-abef-c51a70654104-kube-api-access-5c5nj" pod "698187ec-6d8a-4096-abef-c51a70654104" (UID: "698187ec-6d8a-4096-abef-c51a70654104"). InnerVolumeSpecName "kube-api-access-5c5nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:54.955819 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.955769    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ddd010-79e1-4281-b3ba-a5c0f273fde1-kube-api-access-vtz9h" pod "11ddd010-79e1-4281-b3ba-a5c0f273fde1" (UID: "11ddd010-79e1-4281-b3ba-a5c0f273fde1"). InnerVolumeSpecName "kube-api-access-vtz9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:54.956269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:54.956250    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tls-certs" pod "34cee5d4-7fa9-4857-9916-49ab2a89f8e0" (UID: "34cee5d4-7fa9-4857-9916-49ab2a89f8e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:31:55.050666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050588    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.050666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050622    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.050666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050633    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698187ec-6d8a-4096-abef-c51a70654104-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.050666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050641    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.050666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050649    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11ddd010-79e1-4281-b3ba-a5c0f273fde1-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.050666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050657    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rthnm\" (UniqueName: \"kubernetes.io/projected/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-kube-api-access-rthnm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.050666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050666    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtz9h\" (UniqueName: \"kubernetes.io/projected/11ddd010-79e1-4281-b3ba-a5c0f273fde1-kube-api-access-vtz9h\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.051074 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050675    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11ddd010-79e1-4281-b3ba-a5c0f273fde1-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.051074 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050683    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.051074 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050691    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/698187ec-6d8a-4096-abef-c51a70654104-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.051074 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050699    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.051074 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050707    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.051074 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050714    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c5nj\" (UniqueName: \"kubernetes.io/projected/698187ec-6d8a-4096-abef-c51a70654104-kube-api-access-5c5nj\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.051074 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.050722    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34cee5d4-7fa9-4857-9916-49ab2a89f8e0-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:55.286580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.286547    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"]
Nov 02 08:31:55.287172 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.287157    4898 cpu_manager.go:414] "containerMap: removing container" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main"
Nov 02 08:31:55.287219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.287173    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main"
Nov 02 08:31:55.287219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.287188    4898 cpu_manager.go:414] "containerMap: removing container" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main"
Nov 02 08:31:55.287219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.287193    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main"
Nov 02 08:31:55.287329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.287224    4898 cpu_manager.go:414] "containerMap: removing container" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main"
Nov 02 08:31:55.287329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.287229    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main"
Nov 02 08:31:55.287329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.287239    4898 cpu_manager.go:414] "containerMap: removing container" podUID="acdd4d3a-3be4-4524-8622-217cf531d6cf" containerName="revoke-keys"
Nov 02 08:31:55.287329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.287244    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdd4d3a-3be4-4524-8622-217cf531d6cf" containerName="revoke-keys"
Nov 02 08:31:55.287450 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.287340    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main"
Nov 02 08:31:55.287450 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.287349    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" containerName="main"
Nov 02 08:31:55.287450 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.287363    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main"
Nov 02 08:31:55.287450 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.287370    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdd4d3a-3be4-4524-8622-217cf531d6cf" containerName="revoke-keys"
Nov 02 08:31:55.292635 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.292613    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:55.307512 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.307448    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:31:55.309361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.309333    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"]
Nov 02 08:31:55.315175 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.315144    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:31:55.315282 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.315190    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:31:55.315348 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.315323    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:31:55.455678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.455632    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dc33abe-lhrkm\" (UID: \"04d2d129-9f29-4ba8-8995-5e6442461130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:55.456102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.455763    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dc33abe-lhrkm\" (UID: \"04d2d129-9f29-4ba8-8995-5e6442461130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:55.456102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.456006    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z682k\" (UniqueName: \"kubernetes.io/projected/04d2d129-9f29-4ba8-8995-5e6442461130-kube-api-access-z682k\") pod \"maas-api-revoke-sub-7dc33abe-lhrkm\" (UID: \"04d2d129-9f29-4ba8-8995-5e6442461130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:55.557387 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.557348    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dc33abe-lhrkm\" (UID: \"04d2d129-9f29-4ba8-8995-5e6442461130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:55.558001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.557467    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dc33abe-lhrkm\" (UID: \"04d2d129-9f29-4ba8-8995-5e6442461130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:55.558170 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.558142    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dc33abe-lhrkm\" (UID: \"04d2d129-9f29-4ba8-8995-5e6442461130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:55.558251 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.558179    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z682k\" (UniqueName: \"kubernetes.io/projected/04d2d129-9f29-4ba8-8995-5e6442461130-kube-api-access-z682k\") pod \"maas-api-revoke-sub-7dc33abe-lhrkm\" (UID: \"04d2d129-9f29-4ba8-8995-5e6442461130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:55.560428 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.560399    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dc33abe-lhrkm\" (UID: \"04d2d129-9f29-4ba8-8995-5e6442461130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:55.581739 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.581705    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z682k\" (UniqueName: \"kubernetes.io/projected/04d2d129-9f29-4ba8-8995-5e6442461130-kube-api-access-z682k\") pod \"maas-api-revoke-sub-7dc33abe-lhrkm\" (UID: \"04d2d129-9f29-4ba8-8995-5e6442461130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:55.608367 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.608340    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:55.640821 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.640775    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" podUID="698187ec-6d8a-4096-abef-c51a70654104" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:31:55.641921 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.641852    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:31:55.730983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.730935    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7" event={"ID":"11ddd010-79e1-4281-b3ba-a5c0f273fde1","Type":"ContainerDied","Data":"f12368db957966a8429a19c18b338938ac9ea720e735436b6d4b035e7466986c"}
Nov 02 08:31:55.730983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.730980    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"
Nov 02 08:31:55.731229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.731007    4898 scope.go:122] "RemoveContainer" containerID="429e49c5d6d57d3297caf95da943cc459a1018f4bba61946ce759d109a49a468"
Nov 02 08:31:55.732867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.732847    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"
Nov 02 08:31:55.733012 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.732881    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx" event={"ID":"698187ec-6d8a-4096-abef-c51a70654104","Type":"ContainerDied","Data":"f968b29912bce37a9f9f24eab3817f3f2d573f6623b0b101c2bbcca5de7d8c62"}
Nov 02 08:31:55.734767 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.734750    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"
Nov 02 08:31:55.734938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.734795    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h" event={"ID":"34cee5d4-7fa9-4857-9916-49ab2a89f8e0","Type":"ContainerDied","Data":"b3d5a357644dee547326dbba1f4fd4513686e93820902f615c57719d99761c15"}
Nov 02 08:31:55.745992 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.745970    4898 scope.go:122] "RemoveContainer" containerID="6333e57018fb396231aa307a5f430c9f990f5a44109c5cc34c1eb74bf4509de8"
Nov 02 08:31:55.756190 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.756166    4898 scope.go:122] "RemoveContainer" containerID="c6e808c347ec39d11c99a28a11c1f6b1a2fc93cfb1c3a01778e4ef253a5e12b2"
Nov 02 08:31:55.785272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.785247    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"]
Nov 02 08:31:55.797248 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.797226    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/simulator-w0-0fbc17-kserve-65bd4fb648-gxn8h"]
Nov 02 08:31:55.823846 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.823784    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"]
Nov 02 08:31:55.828303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.828284    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/distinct-w0-0fbc17-kserve-7bd65cdcb4-t49mx"]
Nov 02 08:31:55.864270 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.864245    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"]
Nov 02 08:31:55.867696 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.867675    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0fbc17/premium-w0-0fbc17-kserve-58c44cd85-nrtk7"]
Nov 02 08:31:55.921557 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:31:55.921530    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d2d129_9f29_4ba8_8995_5e6442461130.slice/crio-17054a64ea00424ee778bac5afdc9a7828c831a05b66975ed6414868007751e4 WatchSource:0}: Error finding container 17054a64ea00424ee778bac5afdc9a7828c831a05b66975ed6414868007751e4: Status 404 returned error can't find the container with id 17054a64ea00424ee778bac5afdc9a7828c831a05b66975ed6414868007751e4
Nov 02 08:31:55.921744 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:55.921723    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"]
Nov 02 08:31:56.748881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:56.748837    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm" event={"ID":"04d2d129-9f29-4ba8-8995-5e6442461130","Type":"ContainerStarted","Data":"e12e0b5f83ce858c96c9fb1b47c16a6f5618e92947d3215473ddd2fbec166c12"}
Nov 02 08:31:56.749366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:56.748967    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm" event={"ID":"04d2d129-9f29-4ba8-8995-5e6442461130","Type":"ContainerStarted","Data":"17054a64ea00424ee778bac5afdc9a7828c831a05b66975ed6414868007751e4"}
Nov 02 08:31:57.467319 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:57.467282    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ddd010-79e1-4281-b3ba-a5c0f273fde1" path="/var/lib/kubelet/pods/11ddd010-79e1-4281-b3ba-a5c0f273fde1/volumes"
Nov 02 08:31:57.467838 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:57.467817    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34cee5d4-7fa9-4857-9916-49ab2a89f8e0" path="/var/lib/kubelet/pods/34cee5d4-7fa9-4857-9916-49ab2a89f8e0/volumes"
Nov 02 08:31:57.468308 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:57.468289    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698187ec-6d8a-4096-abef-c51a70654104" path="/var/lib/kubelet/pods/698187ec-6d8a-4096-abef-c51a70654104/volumes"
Nov 02 08:31:57.840468 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:57.840370    4898 generic.go:356] "Generic (PLEG): container finished" podID="04d2d129-9f29-4ba8-8995-5e6442461130" containerID="e12e0b5f83ce858c96c9fb1b47c16a6f5618e92947d3215473ddd2fbec166c12" exitCode=0
Nov 02 08:31:57.842439 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:57.842406    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm" event={"ID":"04d2d129-9f29-4ba8-8995-5e6442461130","Type":"ContainerDied","Data":"e12e0b5f83ce858c96c9fb1b47c16a6f5618e92947d3215473ddd2fbec166c12"}
Nov 02 08:31:59.031623 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.031595    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:59.137019 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.136927    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-openshift-ca\") pod \"04d2d129-9f29-4ba8-8995-5e6442461130\" (UID: \"04d2d129-9f29-4ba8-8995-5e6442461130\") "
Nov 02 08:31:59.137019 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.136972    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-xks-ca\") pod \"04d2d129-9f29-4ba8-8995-5e6442461130\" (UID: \"04d2d129-9f29-4ba8-8995-5e6442461130\") "
Nov 02 08:31:59.137280 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.137131    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d2d129-9f29-4ba8-8995-5e6442461130-kube-api-access-z682k\" (UniqueName: \"kubernetes.io/projected/04d2d129-9f29-4ba8-8995-5e6442461130-kube-api-access-z682k\") pod \"04d2d129-9f29-4ba8-8995-5e6442461130\" (UID: \"04d2d129-9f29-4ba8-8995-5e6442461130\") "
Nov 02 08:31:59.137383 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.137354    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-openshift-ca" pod "04d2d129-9f29-4ba8-8995-5e6442461130" (UID: "04d2d129-9f29-4ba8-8995-5e6442461130"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:31:59.137733 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.137709    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:59.139480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.139446    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-xks-ca" pod "04d2d129-9f29-4ba8-8995-5e6442461130" (UID: "04d2d129-9f29-4ba8-8995-5e6442461130"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:31:59.139969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.139936    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d2d129-9f29-4ba8-8995-5e6442461130-kube-api-access-z682k" pod "04d2d129-9f29-4ba8-8995-5e6442461130" (UID: "04d2d129-9f29-4ba8-8995-5e6442461130"). InnerVolumeSpecName "kube-api-access-z682k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:31:59.238592 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.238545    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d2d129-9f29-4ba8-8995-5e6442461130-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:59.238592 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.238588    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z682k\" (UniqueName: \"kubernetes.io/projected/04d2d129-9f29-4ba8-8995-5e6442461130-kube-api-access-z682k\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:31:59.867848 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.867761    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"
Nov 02 08:31:59.868168 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.867759    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm" event={"ID":"04d2d129-9f29-4ba8-8995-5e6442461130","Type":"ContainerDied","Data":"17054a64ea00424ee778bac5afdc9a7828c831a05b66975ed6414868007751e4"}
Nov 02 08:31:59.868294 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:31:59.868207    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17054a64ea00424ee778bac5afdc9a7828c831a05b66975ed6414868007751e4"
Nov 02 08:32:02.611540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.611504    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"]
Nov 02 08:32:02.612356 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.612334    4898 cpu_manager.go:414] "containerMap: removing container" podUID="04d2d129-9f29-4ba8-8995-5e6442461130" containerName="revoke-keys"
Nov 02 08:32:02.612416 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.612358    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d2d129-9f29-4ba8-8995-5e6442461130" containerName="revoke-keys"
Nov 02 08:32:02.612476 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.612466    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d2d129-9f29-4ba8-8995-5e6442461130" containerName="revoke-keys"
Nov 02 08:32:02.616591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.616566    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:02.632095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.632069    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:32:02.632262 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.632231    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:32:02.635982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.635956    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"]
Nov 02 08:32:02.643827 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.643800    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:32:02.644034 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.643849    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:32:02.700859 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.700825    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeac7505-d4r54\" (UID: \"07d85230-4346-4ab7-a102-2944191e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:02.701021 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.700930    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rv2d\" (UniqueName: \"kubernetes.io/projected/07d85230-4346-4ab7-a102-2944191e0a81-kube-api-access-8rv2d\") pod \"maas-api-revoke-sub-aeac7505-d4r54\" (UID: \"07d85230-4346-4ab7-a102-2944191e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:02.701021 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.700988    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeac7505-d4r54\" (UID: \"07d85230-4346-4ab7-a102-2944191e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:02.802641 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.802583    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeac7505-d4r54\" (UID: \"07d85230-4346-4ab7-a102-2944191e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:02.802839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.802715    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rv2d\" (UniqueName: \"kubernetes.io/projected/07d85230-4346-4ab7-a102-2944191e0a81-kube-api-access-8rv2d\") pod \"maas-api-revoke-sub-aeac7505-d4r54\" (UID: \"07d85230-4346-4ab7-a102-2944191e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:02.802839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.802810    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeac7505-d4r54\" (UID: \"07d85230-4346-4ab7-a102-2944191e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:02.803474 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.803447    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeac7505-d4r54\" (UID: \"07d85230-4346-4ab7-a102-2944191e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:02.805109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.805086    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeac7505-d4r54\" (UID: \"07d85230-4346-4ab7-a102-2944191e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:02.826406 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.826370    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rv2d\" (UniqueName: \"kubernetes.io/projected/07d85230-4346-4ab7-a102-2944191e0a81-kube-api-access-8rv2d\") pod \"maas-api-revoke-sub-aeac7505-d4r54\" (UID: \"07d85230-4346-4ab7-a102-2944191e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:02.932314 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:02.932274    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:03.240102 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:32:03.240066    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d85230_4346_4ab7_a102_2944191e0a81.slice/crio-3f9137a013675c9e512d668755d4ff29409a5700ec901e99968203f921e0997a WatchSource:0}: Error finding container 3f9137a013675c9e512d668755d4ff29409a5700ec901e99968203f921e0997a: Status 404 returned error can't find the container with id 3f9137a013675c9e512d668755d4ff29409a5700ec901e99968203f921e0997a
Nov 02 08:32:03.240290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:03.240272    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"]
Nov 02 08:32:03.932860 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:03.932816    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54" event={"ID":"07d85230-4346-4ab7-a102-2944191e0a81","Type":"ContainerStarted","Data":"a86515ab80e66cdbb046040ac9825b43db427c3bf01af511b875786ed4af562b"}
Nov 02 08:32:03.933360 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:03.932917    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54" event={"ID":"07d85230-4346-4ab7-a102-2944191e0a81","Type":"ContainerStarted","Data":"3f9137a013675c9e512d668755d4ff29409a5700ec901e99968203f921e0997a"}
Nov 02 08:32:03.974826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:03.974765    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54" podStartSLOduration=1.974746705 podStartE2EDuration="1.974746705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:32:03.972181082 +0000 UTC m=+4074.662508047" watchObservedRunningTime="2026-11-02 08:32:03.974746705 +0000 UTC m=+4074.665073671"
Nov 02 08:32:04.372484 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:04.372444    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4" podUID="ea1f4f74-4502-49dd-a54a-893b9264a0cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:32:04.372646 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:04.372509    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5" podUID="c9606134-fab6-48c1-9543-b20f0cf28967" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:32:04.941899 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:04.941847    4898 generic.go:356] "Generic (PLEG): container finished" podID="07d85230-4346-4ab7-a102-2944191e0a81" containerID="a86515ab80e66cdbb046040ac9825b43db427c3bf01af511b875786ed4af562b" exitCode=0
Nov 02 08:32:04.942383 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:04.941944    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54" event={"ID":"07d85230-4346-4ab7-a102-2944191e0a81","Type":"ContainerDied","Data":"a86515ab80e66cdbb046040ac9825b43db427c3bf01af511b875786ed4af562b"}
Nov 02 08:32:05.943693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:05.943657    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"]
Nov 02 08:32:05.948424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:05.948396    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:05.967517 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:05.967486    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"]
Nov 02 08:32:06.048702 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.048666    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc354ad3-5hjjq\" (UID: \"850a7958-bf3e-4176-8e57-29001545b7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:06.048865 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.048844    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc354ad3-5hjjq\" (UID: \"850a7958-bf3e-4176-8e57-29001545b7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:06.049025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.048996    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh7g9\" (UniqueName: \"kubernetes.io/projected/850a7958-bf3e-4176-8e57-29001545b7be-kube-api-access-gh7g9\") pod \"maas-api-revoke-sub-fc354ad3-5hjjq\" (UID: \"850a7958-bf3e-4176-8e57-29001545b7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:06.129276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.129247    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:06.150537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.150511    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc354ad3-5hjjq\" (UID: \"850a7958-bf3e-4176-8e57-29001545b7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:06.150662 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.150606    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc354ad3-5hjjq\" (UID: \"850a7958-bf3e-4176-8e57-29001545b7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:06.150662 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.150648    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh7g9\" (UniqueName: \"kubernetes.io/projected/850a7958-bf3e-4176-8e57-29001545b7be-kube-api-access-gh7g9\") pod \"maas-api-revoke-sub-fc354ad3-5hjjq\" (UID: \"850a7958-bf3e-4176-8e57-29001545b7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:06.151266 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.151247    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc354ad3-5hjjq\" (UID: \"850a7958-bf3e-4176-8e57-29001545b7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:06.152970 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.152943    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc354ad3-5hjjq\" (UID: \"850a7958-bf3e-4176-8e57-29001545b7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:06.176631 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.176602    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh7g9\" (UniqueName: \"kubernetes.io/projected/850a7958-bf3e-4176-8e57-29001545b7be-kube-api-access-gh7g9\") pod \"maas-api-revoke-sub-fc354ad3-5hjjq\" (UID: \"850a7958-bf3e-4176-8e57-29001545b7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:06.251734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.251648    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d85230-4346-4ab7-a102-2944191e0a81-kube-api-access-8rv2d\" (UniqueName: \"kubernetes.io/projected/07d85230-4346-4ab7-a102-2944191e0a81-kube-api-access-8rv2d\") pod \"07d85230-4346-4ab7-a102-2944191e0a81\" (UID: \"07d85230-4346-4ab7-a102-2944191e0a81\") "
Nov 02 08:32:06.251931 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.251741    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-xks-ca\") pod \"07d85230-4346-4ab7-a102-2944191e0a81\" (UID: \"07d85230-4346-4ab7-a102-2944191e0a81\") "
Nov 02 08:32:06.251931 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.251803    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-openshift-ca\") pod \"07d85230-4346-4ab7-a102-2944191e0a81\" (UID: \"07d85230-4346-4ab7-a102-2944191e0a81\") "
Nov 02 08:32:06.252217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.252194    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-openshift-ca" pod "07d85230-4346-4ab7-a102-2944191e0a81" (UID: "07d85230-4346-4ab7-a102-2944191e0a81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:32:06.252425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.252406    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:06.253765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.253743    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d85230-4346-4ab7-a102-2944191e0a81-kube-api-access-8rv2d" pod "07d85230-4346-4ab7-a102-2944191e0a81" (UID: "07d85230-4346-4ab7-a102-2944191e0a81"). InnerVolumeSpecName "kube-api-access-8rv2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:06.253920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.253880    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-xks-ca" pod "07d85230-4346-4ab7-a102-2944191e0a81" (UID: "07d85230-4346-4ab7-a102-2944191e0a81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:32:06.266039 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.266014    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:06.353874 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.353838    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rv2d\" (UniqueName: \"kubernetes.io/projected/07d85230-4346-4ab7-a102-2944191e0a81-kube-api-access-8rv2d\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:06.353874 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.353872    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d85230-4346-4ab7-a102-2944191e0a81-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:06.574548 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:32:06.574509    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850a7958_bf3e_4176_8e57_29001545b7be.slice/crio-970ee61b2a225d5301c9ccb258951b6c8c682f58691989642267985336119414 WatchSource:0}: Error finding container 970ee61b2a225d5301c9ccb258951b6c8c682f58691989642267985336119414: Status 404 returned error can't find the container with id 970ee61b2a225d5301c9ccb258951b6c8c682f58691989642267985336119414
Nov 02 08:32:06.575853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.575828    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"]
Nov 02 08:32:06.959759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.959723    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq" event={"ID":"850a7958-bf3e-4176-8e57-29001545b7be","Type":"ContainerStarted","Data":"188eedbf65a49a6be08c0936bf1ddb207af188079a879eaa2e5905dd6a0c89b5"}
Nov 02 08:32:06.960198 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.959779    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq" event={"ID":"850a7958-bf3e-4176-8e57-29001545b7be","Type":"ContainerStarted","Data":"970ee61b2a225d5301c9ccb258951b6c8c682f58691989642267985336119414"}
Nov 02 08:32:06.961427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.961402    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"
Nov 02 08:32:06.961967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.961440    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54" event={"ID":"07d85230-4346-4ab7-a102-2944191e0a81","Type":"ContainerDied","Data":"3f9137a013675c9e512d668755d4ff29409a5700ec901e99968203f921e0997a"}
Nov 02 08:32:06.961967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:06.961486    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f9137a013675c9e512d668755d4ff29409a5700ec901e99968203f921e0997a"
Nov 02 08:32:07.000383 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:07.000335    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq" podStartSLOduration=2.000321027 podStartE2EDuration="2.000321027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:32:06.999169242 +0000 UTC m=+4077.689496227" watchObservedRunningTime="2026-11-02 08:32:07.000321027 +0000 UTC m=+4077.690647991"
Nov 02 08:32:07.970387 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:07.970340    4898 generic.go:356] "Generic (PLEG): container finished" podID="850a7958-bf3e-4176-8e57-29001545b7be" containerID="188eedbf65a49a6be08c0936bf1ddb207af188079a879eaa2e5905dd6a0c89b5" exitCode=0
Nov 02 08:32:07.970862 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:07.970423    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq" event={"ID":"850a7958-bf3e-4176-8e57-29001545b7be","Type":"ContainerDied","Data":"188eedbf65a49a6be08c0936bf1ddb207af188079a879eaa2e5905dd6a0c89b5"}
Nov 02 08:32:09.166665 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.166633    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:09.295018 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.294922    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850a7958-bf3e-4176-8e57-29001545b7be-kube-api-access-gh7g9\" (UniqueName: \"kubernetes.io/projected/850a7958-bf3e-4176-8e57-29001545b7be-kube-api-access-gh7g9\") pod \"850a7958-bf3e-4176-8e57-29001545b7be\" (UID: \"850a7958-bf3e-4176-8e57-29001545b7be\") "
Nov 02 08:32:09.295018 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.294985    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-xks-ca\") pod \"850a7958-bf3e-4176-8e57-29001545b7be\" (UID: \"850a7958-bf3e-4176-8e57-29001545b7be\") "
Nov 02 08:32:09.295264 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.295029    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-openshift-ca\") pod \"850a7958-bf3e-4176-8e57-29001545b7be\" (UID: \"850a7958-bf3e-4176-8e57-29001545b7be\") "
Nov 02 08:32:09.305930 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.296092    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-openshift-ca" pod "850a7958-bf3e-4176-8e57-29001545b7be" (UID: "850a7958-bf3e-4176-8e57-29001545b7be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:32:09.305930 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.297074    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:09.314556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.314523    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-xks-ca" pod "850a7958-bf3e-4176-8e57-29001545b7be" (UID: "850a7958-bf3e-4176-8e57-29001545b7be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:32:09.314740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.314720    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850a7958-bf3e-4176-8e57-29001545b7be-kube-api-access-gh7g9" pod "850a7958-bf3e-4176-8e57-29001545b7be" (UID: "850a7958-bf3e-4176-8e57-29001545b7be"). InnerVolumeSpecName "kube-api-access-gh7g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:09.397853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.397819    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850a7958-bf3e-4176-8e57-29001545b7be-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:09.397853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.397850    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh7g9\" (UniqueName: \"kubernetes.io/projected/850a7958-bf3e-4176-8e57-29001545b7be-kube-api-access-gh7g9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:09.990046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.990010    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq" event={"ID":"850a7958-bf3e-4176-8e57-29001545b7be","Type":"ContainerDied","Data":"970ee61b2a225d5301c9ccb258951b6c8c682f58691989642267985336119414"}
Nov 02 08:32:09.990205 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.990065    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="970ee61b2a225d5301c9ccb258951b6c8c682f58691989642267985336119414"
Nov 02 08:32:09.990205 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:09.990027    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"
Nov 02 08:32:12.889604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:12.889571    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"
Nov 02 08:32:12.956065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:12.956014    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"
Nov 02 08:32:13.021014 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.020979    4898 generic.go:356] "Generic (PLEG): container finished" podID="ea1f4f74-4502-49dd-a54a-893b9264a0cf" containerID="1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8" exitCode=0
Nov 02 08:32:13.021221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.021044    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"
Nov 02 08:32:13.021221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.021074    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4" event={"ID":"ea1f4f74-4502-49dd-a54a-893b9264a0cf","Type":"ContainerDied","Data":"1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8"}
Nov 02 08:32:13.021221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.021138    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4" event={"ID":"ea1f4f74-4502-49dd-a54a-893b9264a0cf","Type":"ContainerDied","Data":"65ceb3875cc4f48d8b0b4cb5fcf2dbe2cf7e9a483b7765c542a4c7517981aa9a"}
Nov 02 08:32:13.021221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.021170    4898 scope.go:122] "RemoveContainer" containerID="1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8"
Nov 02 08:32:13.023097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.023029    4898 generic.go:356] "Generic (PLEG): container finished" podID="c9606134-fab6-48c1-9543-b20f0cf28967" containerID="e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24" exitCode=0
Nov 02 08:32:13.023097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.023056    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5" event={"ID":"c9606134-fab6-48c1-9543-b20f0cf28967","Type":"ContainerDied","Data":"e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24"}
Nov 02 08:32:13.023097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.023084    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"
Nov 02 08:32:13.023317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.023112    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5" event={"ID":"c9606134-fab6-48c1-9543-b20f0cf28967","Type":"ContainerDied","Data":"790b37aa5ed24021e497a87dc90c3f8295326b0c0c949f06524e21e59561d604"}
Nov 02 08:32:13.034972 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.034944    4898 scope.go:122] "RemoveContainer" containerID="1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8"
Nov 02 08:32:13.035304 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:32:13.035281    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8\": container with ID starting with 1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8 not found: ID does not exist" containerID="1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8"
Nov 02 08:32:13.035404 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.035314    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8"} err="failed to get container status \"1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8\": rpc error: code = NotFound desc = could not find container \"1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8\": container with ID starting with 1cb3c37016a066acaeee7cce318d360225500e70e639bb14adf8d27656f2fdd8 not found: ID does not exist"
Nov 02 08:32:13.035404 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.035333    4898 scope.go:122] "RemoveContainer" containerID="e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24"
Nov 02 08:32:13.047121 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.047097    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea1f4f74-4502-49dd-a54a-893b9264a0cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1f4f74-4502-49dd-a54a-893b9264a0cf-plugins-config-volume\") pod \"ea1f4f74-4502-49dd-a54a-893b9264a0cf\" (UID: \"ea1f4f74-4502-49dd-a54a-893b9264a0cf\") "
Nov 02 08:32:13.047244 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.047215    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea1f4f74-4502-49dd-a54a-893b9264a0cf-kube-api-access-q8vf9\" (UniqueName: \"kubernetes.io/projected/ea1f4f74-4502-49dd-a54a-893b9264a0cf-kube-api-access-q8vf9\") pod \"ea1f4f74-4502-49dd-a54a-893b9264a0cf\" (UID: \"ea1f4f74-4502-49dd-a54a-893b9264a0cf\") "
Nov 02 08:32:13.047316 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.047265    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9606134-fab6-48c1-9543-b20f0cf28967-kube-api-access-2qrpn\" (UniqueName: \"kubernetes.io/projected/c9606134-fab6-48c1-9543-b20f0cf28967-kube-api-access-2qrpn\") pod \"c9606134-fab6-48c1-9543-b20f0cf28967\" (UID: \"c9606134-fab6-48c1-9543-b20f0cf28967\") "
Nov 02 08:32:13.047415 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.047396    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9606134-fab6-48c1-9543-b20f0cf28967-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9606134-fab6-48c1-9543-b20f0cf28967-plugins-config-volume\") pod \"c9606134-fab6-48c1-9543-b20f0cf28967\" (UID: \"c9606134-fab6-48c1-9543-b20f0cf28967\") "
Nov 02 08:32:13.047530 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.047512    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea1f4f74-4502-49dd-a54a-893b9264a0cf-plugins-config-volume" pod "ea1f4f74-4502-49dd-a54a-893b9264a0cf" (UID: "ea1f4f74-4502-49dd-a54a-893b9264a0cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:32:13.047776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.047752    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9606134-fab6-48c1-9543-b20f0cf28967-plugins-config-volume" pod "c9606134-fab6-48c1-9543-b20f0cf28967" (UID: "c9606134-fab6-48c1-9543-b20f0cf28967"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:32:13.047998 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.047977    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1f4f74-4502-49dd-a54a-893b9264a0cf-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:13.048109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.048001    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9606134-fab6-48c1-9543-b20f0cf28967-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:13.048109 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.048002    4898 scope.go:122] "RemoveContainer" containerID="e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24"
Nov 02 08:32:13.048335 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:32:13.048311    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24\": container with ID starting with e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24 not found: ID does not exist" containerID="e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24"
Nov 02 08:32:13.048400 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.048341    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24"} err="failed to get container status \"e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24\": rpc error: code = NotFound desc = could not find container \"e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24\": container with ID starting with e286b9a04d25a27ec18f611e96d238ac37c9bc68b115bfa8d8bcc8d216b7dc24 not found: ID does not exist"
Nov 02 08:32:13.049442 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.049421    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1f4f74-4502-49dd-a54a-893b9264a0cf-kube-api-access-q8vf9" pod "ea1f4f74-4502-49dd-a54a-893b9264a0cf" (UID: "ea1f4f74-4502-49dd-a54a-893b9264a0cf"). InnerVolumeSpecName "kube-api-access-q8vf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:13.049831 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.049811    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9606134-fab6-48c1-9543-b20f0cf28967-kube-api-access-2qrpn" pod "c9606134-fab6-48c1-9543-b20f0cf28967" (UID: "c9606134-fab6-48c1-9543-b20f0cf28967"). InnerVolumeSpecName "kube-api-access-2qrpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:13.149708 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.149606    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8vf9\" (UniqueName: \"kubernetes.io/projected/ea1f4f74-4502-49dd-a54a-893b9264a0cf-kube-api-access-q8vf9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:13.149708 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.149651    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qrpn\" (UniqueName: \"kubernetes.io/projected/c9606134-fab6-48c1-9543-b20f0cf28967-kube-api-access-2qrpn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:13.370164 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.370129    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"]
Nov 02 08:32:13.382292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.382261    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0fbc17-69fd8fbf59-r54s4"]
Nov 02 08:32:13.416975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.416942    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"]
Nov 02 08:32:13.420488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.420467    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0fbc17-849c8fb697-82cv5"]
Nov 02 08:32:13.466474 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.466433    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9606134-fab6-48c1-9543-b20f0cf28967" path="/var/lib/kubelet/pods/c9606134-fab6-48c1-9543-b20f0cf28967/volumes"
Nov 02 08:32:13.467076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.467052    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea1f4f74-4502-49dd-a54a-893b9264a0cf" path="/var/lib/kubelet/pods/ea1f4f74-4502-49dd-a54a-893b9264a0cf/volumes"
Nov 02 08:32:13.475014 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.474990    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"]
Nov 02 08:32:13.475862 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.475843    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c9606134-fab6-48c1-9543-b20f0cf28967" containerName="payload-processing"
Nov 02 08:32:13.475948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.475867    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9606134-fab6-48c1-9543-b20f0cf28967" containerName="payload-processing"
Nov 02 08:32:13.475948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.475917    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ea1f4f74-4502-49dd-a54a-893b9264a0cf" containerName="payload-pre-processing"
Nov 02 08:32:13.475948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.475926    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea1f4f74-4502-49dd-a54a-893b9264a0cf" containerName="payload-pre-processing"
Nov 02 08:32:13.475948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.475941    4898 cpu_manager.go:414] "containerMap: removing container" podUID="07d85230-4346-4ab7-a102-2944191e0a81" containerName="revoke-keys"
Nov 02 08:32:13.476094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.475949    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d85230-4346-4ab7-a102-2944191e0a81" containerName="revoke-keys"
Nov 02 08:32:13.476094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.475978    4898 cpu_manager.go:414] "containerMap: removing container" podUID="850a7958-bf3e-4176-8e57-29001545b7be" containerName="revoke-keys"
Nov 02 08:32:13.476094 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.475985    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850a7958-bf3e-4176-8e57-29001545b7be" containerName="revoke-keys"
Nov 02 08:32:13.476209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.476123    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9606134-fab6-48c1-9543-b20f0cf28967" containerName="payload-processing"
Nov 02 08:32:13.476209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.476141    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea1f4f74-4502-49dd-a54a-893b9264a0cf" containerName="payload-pre-processing"
Nov 02 08:32:13.476209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.476157    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d85230-4346-4ab7-a102-2944191e0a81" containerName="revoke-keys"
Nov 02 08:32:13.476209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.476171    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="850a7958-bf3e-4176-8e57-29001545b7be" containerName="revoke-keys"
Nov 02 08:32:13.481705 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.481686    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:13.497362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.496520    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"]
Nov 02 08:32:13.497362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.496780    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:32:13.497362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.496859    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:32:13.497362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.496997    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:32:13.497362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.497047    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:32:13.560697 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.560658    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb475eb-jwcxh\" (UID: \"29227432-f91a-4c80-9010-8c8c313a145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:13.560902 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.560799    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkmhk\" (UniqueName: \"kubernetes.io/projected/29227432-f91a-4c80-9010-8c8c313a145c-kube-api-access-bkmhk\") pod \"maas-api-revoke-sub-5fb475eb-jwcxh\" (UID: \"29227432-f91a-4c80-9010-8c8c313a145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:13.560959 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.560911    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb475eb-jwcxh\" (UID: \"29227432-f91a-4c80-9010-8c8c313a145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:13.661876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.661836    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb475eb-jwcxh\" (UID: \"29227432-f91a-4c80-9010-8c8c313a145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:13.662068 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.661956    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkmhk\" (UniqueName: \"kubernetes.io/projected/29227432-f91a-4c80-9010-8c8c313a145c-kube-api-access-bkmhk\") pod \"maas-api-revoke-sub-5fb475eb-jwcxh\" (UID: \"29227432-f91a-4c80-9010-8c8c313a145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:13.662068 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.661997    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb475eb-jwcxh\" (UID: \"29227432-f91a-4c80-9010-8c8c313a145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:13.662717 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.662682    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb475eb-jwcxh\" (UID: \"29227432-f91a-4c80-9010-8c8c313a145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:13.664581 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.664553    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb475eb-jwcxh\" (UID: \"29227432-f91a-4c80-9010-8c8c313a145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:13.692365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.692303    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkmhk\" (UniqueName: \"kubernetes.io/projected/29227432-f91a-4c80-9010-8c8c313a145c-kube-api-access-bkmhk\") pod \"maas-api-revoke-sub-5fb475eb-jwcxh\" (UID: \"29227432-f91a-4c80-9010-8c8c313a145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:13.783687 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.783649    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"]
Nov 02 08:32:13.800833 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.800797    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:13.821106 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:13.821080    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"]
Nov 02 08:32:14.231842 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:32:14.231802    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29227432_f91a_4c80_9010_8c8c313a145c.slice/crio-5b51102745942a69044e6c8c6b7600af8177b118df28bb2972049f3655757805 WatchSource:0}: Error finding container 5b51102745942a69044e6c8c6b7600af8177b118df28bb2972049f3655757805: Status 404 returned error can't find the container with id 5b51102745942a69044e6c8c6b7600af8177b118df28bb2972049f3655757805
Nov 02 08:32:14.233579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:14.233549    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"]
Nov 02 08:32:14.996290 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:32:14.996213    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:32:15.047267 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:15.047226    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh" event={"ID":"29227432-f91a-4c80-9010-8c8c313a145c","Type":"ContainerStarted","Data":"07a7587e751c01b2e84b4da9e66aa952595929b283c57f1c41b9a6cc99750a7b"}
Nov 02 08:32:15.047430 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:15.047302    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh" event={"ID":"29227432-f91a-4c80-9010-8c8c313a145c","Type":"ContainerStarted","Data":"5b51102745942a69044e6c8c6b7600af8177b118df28bb2972049f3655757805"}
Nov 02 08:32:15.100290 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:15.100214    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh" podStartSLOduration=2.100195161 podStartE2EDuration="2.100195161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:32:15.096753461 +0000 UTC m=+4085.787080425" watchObservedRunningTime="2026-11-02 08:32:15.100195161 +0000 UTC m=+4085.790522125"
Nov 02 08:32:15.200784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:15.200739    4898 scope.go:122] "RemoveContainer" containerID="98c3263aa7ffea54d982adf9a762b576aaa3735ccb288295ae3dd19f79818a6b"
Nov 02 08:32:15.211575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:15.211547    4898 scope.go:122] "RemoveContainer" containerID="3006b406edaccc710fc279a9ae804652c1819c634fda769cadf12dcc0a8b545d"
Nov 02 08:32:15.224300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:15.224269    4898 scope.go:122] "RemoveContainer" containerID="4c0bfff422b788fa70123084981cc58c62691dac218cafd6657800aa5bfe0a40"
Nov 02 08:32:15.242444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:15.242170    4898 scope.go:122] "RemoveContainer" containerID="737e70147b57f6c9aa7c5673ee2bfd36574ffca4bedc7cab48d4cfeae0ae504b"
Nov 02 08:32:15.258874 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:15.258849    4898 scope.go:122] "RemoveContainer" containerID="2651bc06dec826050a74ba8d80e73546f3bfeb7d8301aa3576da0219468fcb2e"
Nov 02 08:32:16.071934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:16.068946    4898 generic.go:356] "Generic (PLEG): container finished" podID="29227432-f91a-4c80-9010-8c8c313a145c" containerID="07a7587e751c01b2e84b4da9e66aa952595929b283c57f1c41b9a6cc99750a7b" exitCode=0
Nov 02 08:32:16.071934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:16.069223    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh" event={"ID":"29227432-f91a-4c80-9010-8c8c313a145c","Type":"ContainerDied","Data":"07a7587e751c01b2e84b4da9e66aa952595929b283c57f1c41b9a6cc99750a7b"}
Nov 02 08:32:17.500011 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:17.495210    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:17.542169 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:17.542127    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-openshift-ca\") pod \"29227432-f91a-4c80-9010-8c8c313a145c\" (UID: \"29227432-f91a-4c80-9010-8c8c313a145c\") "
Nov 02 08:32:17.542354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:17.542194    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29227432-f91a-4c80-9010-8c8c313a145c-kube-api-access-bkmhk\" (UniqueName: \"kubernetes.io/projected/29227432-f91a-4c80-9010-8c8c313a145c-kube-api-access-bkmhk\") pod \"29227432-f91a-4c80-9010-8c8c313a145c\" (UID: \"29227432-f91a-4c80-9010-8c8c313a145c\") "
Nov 02 08:32:17.542354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:17.542263    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-xks-ca\") pod \"29227432-f91a-4c80-9010-8c8c313a145c\" (UID: \"29227432-f91a-4c80-9010-8c8c313a145c\") "
Nov 02 08:32:17.542525 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:17.542497    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-openshift-ca" pod "29227432-f91a-4c80-9010-8c8c313a145c" (UID: "29227432-f91a-4c80-9010-8c8c313a145c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:32:17.542868 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:17.542841    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:17.545090 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:17.545064    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-xks-ca" pod "29227432-f91a-4c80-9010-8c8c313a145c" (UID: "29227432-f91a-4c80-9010-8c8c313a145c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:32:17.545830 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:17.545802    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29227432-f91a-4c80-9010-8c8c313a145c-kube-api-access-bkmhk" pod "29227432-f91a-4c80-9010-8c8c313a145c" (UID: "29227432-f91a-4c80-9010-8c8c313a145c"). InnerVolumeSpecName "kube-api-access-bkmhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:17.643765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:17.643724    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkmhk\" (UniqueName: \"kubernetes.io/projected/29227432-f91a-4c80-9010-8c8c313a145c-kube-api-access-bkmhk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:17.643765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:17.643754    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29227432-f91a-4c80-9010-8c8c313a145c-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:18.136032 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:18.135986    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh" event={"ID":"29227432-f91a-4c80-9010-8c8c313a145c","Type":"ContainerDied","Data":"5b51102745942a69044e6c8c6b7600af8177b118df28bb2972049f3655757805"}
Nov 02 08:32:18.136252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:18.136051    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b51102745942a69044e6c8c6b7600af8177b118df28bb2972049f3655757805"
Nov 02 08:32:18.136252 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:18.136137    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"
Nov 02 08:32:18.795122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:18.787959    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd" podUID="35a88092-f516-49ce-9448-0f5a01ba4174" containerName="payload-processing" containerID="cri-o://21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37" gracePeriod=25
Nov 02 08:32:18.825644 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:18.824708    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs" podUID="266b1f5a-899e-4175-9358-de87e3f7502a" containerName="payload-pre-processing" containerID="cri-o://f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584" gracePeriod=25
Nov 02 08:32:22.329054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:22.328210    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"]
Nov 02 08:32:22.329054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:22.328513    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerName="istio-proxy" containerID="cri-o://7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c" gracePeriod=30
Nov 02 08:32:24.611613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:24.611573    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs" podUID="266b1f5a-899e-4175-9358-de87e3f7502a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:32:24.612061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:24.611740    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd" podUID="35a88092-f516-49ce-9448-0f5a01ba4174" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:32:26.335003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:26.333708    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": read tcp 10.132.0.2:34778->10.132.1.143:15021: read: connection reset by peer" start-of-body=
Nov 02 08:32:26.335003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:26.333779    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": read tcp 10.132.0.2:34778->10.132.1.143:15021: read: connection reset by peer"
Nov 02 08:32:27.279630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.279590    4898 generic.go:356] "Generic (PLEG): container finished" podID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerID="c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973" exitCode=0
Nov 02 08:32:27.279816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.279646    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" event={"ID":"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f","Type":"ContainerDied","Data":"c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973"}
Nov 02 08:32:27.280235 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.280216    4898 scope.go:122] "RemoveContainer" containerID="c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973"
Nov 02 08:32:27.282150 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.282118    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:32:27.282236 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.282192    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:32:27.282236 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.282229    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:32:27.648592 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.648561    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 08:32:27.828133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828046    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-credential-socket\") pod \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") "
Nov 02 08:32:27.828133 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828098    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istiod-ca-cert\") pod \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") "
Nov 02 08:32:27.828396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828192    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-socket\") pod \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") "
Nov 02 08:32:27.828396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828224    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-certs\") pod \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") "
Nov 02 08:32:27.828396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828309    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-envoy\") pod \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") "
Nov 02 08:32:27.828396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828338    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-data\") pod \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") "
Nov 02 08:32:27.828396 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828369    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-podinfo\") pod \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") "
Nov 02 08:32:27.828698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828410    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-credential-socket" pod "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" (UID: "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:27.828698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828431    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-token\" (UniqueName: \"kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-token\") pod \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") "
Nov 02 08:32:27.828698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828538    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-kube-api-access-mfvvk\" (UniqueName: \"kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-kube-api-access-mfvvk\") pod \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\" (UID: \"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e\") "
Nov 02 08:32:27.828698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828638    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-socket" pod "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" (UID: "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:27.828698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828671    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-certs" pod "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" (UID: "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:27.828698 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828691    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-data" pod "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" (UID: "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:27.828991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.828694    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istiod-ca-cert" pod "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" (UID: "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:32:27.829045 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.829029    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:27.829102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.829045    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:27.829102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.829055    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:27.829102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.829068    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:27.829102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.829079    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:27.830824 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.830795    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-envoy" pod "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" (UID: "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:27.831007 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.830827    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-kube-api-access-mfvvk" pod "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" (UID: "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e"). InnerVolumeSpecName "kube-api-access-mfvvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:27.831007 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.830830    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-podinfo" pod "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" (UID: "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:32:27.831007 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.830915    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-token" pod "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" (UID: "e6d217e2-96c8-4040-81d3-b24fdf9f5f2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:27.930473 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.930429    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfvvk\" (UniqueName: \"kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-kube-api-access-mfvvk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:27.930473 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.930465    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:27.930473 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.930479    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:27.930767 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:27.930490    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:28.294780 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.294731    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" event={"ID":"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f","Type":"ContainerStarted","Data":"b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b"}
Nov 02 08:32:28.297037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.297002    4898 generic.go:356] "Generic (PLEG): container finished" podID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerID="7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c" exitCode=0
Nov 02 08:32:28.297203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.297068    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"
Nov 02 08:32:28.297203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.297077    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" event={"ID":"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e","Type":"ContainerDied","Data":"7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c"}
Nov 02 08:32:28.297203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.297127    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4" event={"ID":"e6d217e2-96c8-4040-81d3-b24fdf9f5f2e","Type":"ContainerDied","Data":"1ff2fde2dbbf99dcf7be1bacf5ee6f6db70dcdc5e18051d4ceeb6e990f659c02"}
Nov 02 08:32:28.297203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.297167    4898 scope.go:122] "RemoveContainer" containerID="7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c"
Nov 02 08:32:28.315486 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.315466    4898 scope.go:122] "RemoveContainer" containerID="dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d"
Nov 02 08:32:28.325374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.325348    4898 scope.go:122] "RemoveContainer" containerID="7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c"
Nov 02 08:32:28.325625 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:32:28.325604    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c\": container with ID starting with 7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c not found: ID does not exist" containerID="7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c"
Nov 02 08:32:28.325678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.325639    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c"} err="failed to get container status \"7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c\": rpc error: code = NotFound desc = could not find container \"7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c\": container with ID starting with 7d6d75aa4d2c4a625836f4d0c20f0d1675a61f8d115023a7feab8af2cd18bb5c not found: ID does not exist"
Nov 02 08:32:28.325678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.325659    4898 scope.go:122] "RemoveContainer" containerID="dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d"
Nov 02 08:32:28.325881 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:32:28.325867    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d\": container with ID starting with dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d not found: ID does not exist" containerID="dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d"
Nov 02 08:32:28.325951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.325898    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d"} err="failed to get container status \"dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d\": rpc error: code = NotFound desc = could not find container \"dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d\": container with ID starting with dcbfadee0723a61e076d700360a7c4d437ac78fd025e6eb26cad345b6b04455d not found: ID does not exist"
Nov 02 08:32:28.400393 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.400351    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"]
Nov 02 08:32:28.415476 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:28.415438    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0fbc17-openshift-default-5c4c87fd67-qz8r4"]
Nov 02 08:32:29.039701 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:29.039641    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:32:29.040202 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:29.039930    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body=
Nov 02 08:32:29.040202 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:29.039991    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused"
Nov 02 08:32:29.493098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:29.493059    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" path="/var/lib/kubelet/pods/e6d217e2-96c8-4040-81d3-b24fdf9f5f2e/volumes"
Nov 02 08:32:30.037596 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:30.037555    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body=
Nov 02 08:32:30.037788 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:30.037614    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused"
Nov 02 08:32:31.037668 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:31.037628    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body=
Nov 02 08:32:31.038123 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:31.037691    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused"
Nov 02 08:32:32.037503 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:32.037464    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body=
Nov 02 08:32:32.037685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:32.037534    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused"
Nov 02 08:32:33.037068 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:33.037032    4898 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body=
Nov 02 08:32:33.037467 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:33.037089    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused"
Nov 02 08:32:34.041816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:34.041781    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:32:34.042236 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:34.042181    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:32:34.042994 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:34.042975    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:32:34.611472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:34.611435    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs" podUID="266b1f5a-899e-4175-9358-de87e3f7502a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:32:34.611685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:34.611438    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd" podUID="35a88092-f516-49ce-9448-0f5a01ba4174" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:32:39.102282 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.102248    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"
Nov 02 08:32:39.150158 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.150099    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"
Nov 02 08:32:39.184413 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.184381    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a88092-f516-49ce-9448-0f5a01ba4174-kube-api-access-j58s5\" (UniqueName: \"kubernetes.io/projected/35a88092-f516-49ce-9448-0f5a01ba4174-kube-api-access-j58s5\") pod \"35a88092-f516-49ce-9448-0f5a01ba4174\" (UID: \"35a88092-f516-49ce-9448-0f5a01ba4174\") "
Nov 02 08:32:39.184587 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.184535    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a88092-f516-49ce-9448-0f5a01ba4174-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35a88092-f516-49ce-9448-0f5a01ba4174-plugins-config-volume\") pod \"35a88092-f516-49ce-9448-0f5a01ba4174\" (UID: \"35a88092-f516-49ce-9448-0f5a01ba4174\") "
Nov 02 08:32:39.184669 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.184587    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266b1f5a-899e-4175-9358-de87e3f7502a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266b1f5a-899e-4175-9358-de87e3f7502a-plugins-config-volume\") pod \"266b1f5a-899e-4175-9358-de87e3f7502a\" (UID: \"266b1f5a-899e-4175-9358-de87e3f7502a\") "
Nov 02 08:32:39.185222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.184746    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266b1f5a-899e-4175-9358-de87e3f7502a-kube-api-access-xlwxq\" (UniqueName: \"kubernetes.io/projected/266b1f5a-899e-4175-9358-de87e3f7502a-kube-api-access-xlwxq\") pod \"266b1f5a-899e-4175-9358-de87e3f7502a\" (UID: \"266b1f5a-899e-4175-9358-de87e3f7502a\") "
Nov 02 08:32:39.185222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.185144    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a88092-f516-49ce-9448-0f5a01ba4174-plugins-config-volume" pod "35a88092-f516-49ce-9448-0f5a01ba4174" (UID: "35a88092-f516-49ce-9448-0f5a01ba4174"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:32:39.185222 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.185181    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266b1f5a-899e-4175-9358-de87e3f7502a-plugins-config-volume" pod "266b1f5a-899e-4175-9358-de87e3f7502a" (UID: "266b1f5a-899e-4175-9358-de87e3f7502a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:32:39.185455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.185425    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35a88092-f516-49ce-9448-0f5a01ba4174-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:39.185455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.185444    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266b1f5a-899e-4175-9358-de87e3f7502a-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:39.187262 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.187237    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a88092-f516-49ce-9448-0f5a01ba4174-kube-api-access-j58s5" pod "35a88092-f516-49ce-9448-0f5a01ba4174" (UID: "35a88092-f516-49ce-9448-0f5a01ba4174"). InnerVolumeSpecName "kube-api-access-j58s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:39.187547 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.187527    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266b1f5a-899e-4175-9358-de87e3f7502a-kube-api-access-xlwxq" pod "266b1f5a-899e-4175-9358-de87e3f7502a" (UID: "266b1f5a-899e-4175-9358-de87e3f7502a"). InnerVolumeSpecName "kube-api-access-xlwxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:39.286746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.286713    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlwxq\" (UniqueName: \"kubernetes.io/projected/266b1f5a-899e-4175-9358-de87e3f7502a-kube-api-access-xlwxq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:39.286746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.286743    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j58s5\" (UniqueName: \"kubernetes.io/projected/35a88092-f516-49ce-9448-0f5a01ba4174-kube-api-access-j58s5\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:39.406860 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.406822    4898 generic.go:356] "Generic (PLEG): container finished" podID="35a88092-f516-49ce-9448-0f5a01ba4174" containerID="21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37" exitCode=0
Nov 02 08:32:39.407095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.406879    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"
Nov 02 08:32:39.407095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.406910    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd" event={"ID":"35a88092-f516-49ce-9448-0f5a01ba4174","Type":"ContainerDied","Data":"21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37"}
Nov 02 08:32:39.407095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.406980    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd" event={"ID":"35a88092-f516-49ce-9448-0f5a01ba4174","Type":"ContainerDied","Data":"f2981b6f0997a7988de93a3b3ac325555c8d5e76711ae780970831d50c6cb86a"}
Nov 02 08:32:39.407095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.407013    4898 scope.go:122] "RemoveContainer" containerID="21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37"
Nov 02 08:32:39.409027 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.408998    4898 generic.go:356] "Generic (PLEG): container finished" podID="266b1f5a-899e-4175-9358-de87e3f7502a" containerID="f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584" exitCode=0
Nov 02 08:32:39.409148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.409054    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"
Nov 02 08:32:39.409148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.409092    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs" event={"ID":"266b1f5a-899e-4175-9358-de87e3f7502a","Type":"ContainerDied","Data":"f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584"}
Nov 02 08:32:39.409229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.409160    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs" event={"ID":"266b1f5a-899e-4175-9358-de87e3f7502a","Type":"ContainerDied","Data":"aa58087573376f0049cdf485a3955d6cfda893d3dcc3bf9f0a60035f563bb22c"}
Nov 02 08:32:39.419533 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.419510    4898 scope.go:122] "RemoveContainer" containerID="21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37"
Nov 02 08:32:39.419822 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:32:39.419798    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37\": container with ID starting with 21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37 not found: ID does not exist" containerID="21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37"
Nov 02 08:32:39.419883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.419830    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37"} err="failed to get container status \"21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37\": rpc error: code = NotFound desc = could not find container \"21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37\": container with ID starting with 21125bd1ea7e1b77ff5d93f85fdcd06000941dd2a15dad6721cc9d0617838f37 not found: ID does not exist"
Nov 02 08:32:39.419883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.419859    4898 scope.go:122] "RemoveContainer" containerID="f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584"
Nov 02 08:32:39.429952 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.429932    4898 scope.go:122] "RemoveContainer" containerID="f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584"
Nov 02 08:32:39.430208 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:32:39.430189    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584\": container with ID starting with f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584 not found: ID does not exist" containerID="f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584"
Nov 02 08:32:39.430272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.430214    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584"} err="failed to get container status \"f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584\": rpc error: code = NotFound desc = could not find container \"f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584\": container with ID starting with f7cc961e04a934bdc1e75ee5cfa43b353be99e7e03ff15823553fd202a0bd584 not found: ID does not exist"
Nov 02 08:32:39.466193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.466170    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"]
Nov 02 08:32:39.469038 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.468990    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7e7a83-7d4bbb4cff-4w5dd"]
Nov 02 08:32:39.532162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.531590    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"]
Nov 02 08:32:39.536930 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:39.534270    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7e7a83-954cf6f9-lpxhs"]
Nov 02 08:32:41.467064 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:41.467027    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="266b1f5a-899e-4175-9358-de87e3f7502a" path="/var/lib/kubelet/pods/266b1f5a-899e-4175-9358-de87e3f7502a/volumes"
Nov 02 08:32:41.467578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:41.467500    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a88092-f516-49ce-9448-0f5a01ba4174" path="/var/lib/kubelet/pods/35a88092-f516-49ce-9448-0f5a01ba4174/volumes"
Nov 02 08:32:44.991912 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:32:44.991268    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:32:47.301708 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:47.301665    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"]
Nov 02 08:32:47.302161 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:47.301958    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerName="istio-proxy" containerID="cri-o://bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5" gracePeriod=30
Nov 02 08:32:49.033972 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:49.033929    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"]
Nov 02 08:32:49.044955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:49.044921    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21c098c-45s6h"]
Nov 02 08:32:49.467272 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:49.467231    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048db14a-1ff5-4e5b-84f1-6b8ab2160e7e" path="/var/lib/kubelet/pods/048db14a-1ff5-4e5b-84f1-6b8ab2160e7e/volumes"
Nov 02 08:32:51.716985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:51.716946    4898 patch_prober.go:29] interesting pod/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:32:51.717386 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:51.717013    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:32:52.586695 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.586622    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 08:32:52.594916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.594872    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-credential-socket\") pod \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") "
Nov 02 08:32:52.595084 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.594996    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-envoy\") pod \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") "
Nov 02 08:32:52.595084 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.595018    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-certs\") pod \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") "
Nov 02 08:32:52.595084 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.595060    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-token\" (UniqueName: \"kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-token\") pod \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") "
Nov 02 08:32:52.595284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.595133    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-data\") pod \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") "
Nov 02 08:32:52.595284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.595142    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-credential-socket" pod "eb5377b5-78d2-4523-b7cc-6eb386e15ad5" (UID: "eb5377b5-78d2-4523-b7cc-6eb386e15ad5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:52.595284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.595160    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-socket\") pod \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") "
Nov 02 08:32:52.595284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.595227    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-podinfo\") pod \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") "
Nov 02 08:32:52.595284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.595266    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istiod-ca-cert\") pod \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") "
Nov 02 08:32:52.595572 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.595312    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-kube-api-access-brl85\" (UniqueName: \"kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-kube-api-access-brl85\") pod \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\" (UID: \"eb5377b5-78d2-4523-b7cc-6eb386e15ad5\") "
Nov 02 08:32:52.595572 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.595532    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-socket" pod "eb5377b5-78d2-4523-b7cc-6eb386e15ad5" (UID: "eb5377b5-78d2-4523-b7cc-6eb386e15ad5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:52.596033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.595853    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-data" pod "eb5377b5-78d2-4523-b7cc-6eb386e15ad5" (UID: "eb5377b5-78d2-4523-b7cc-6eb386e15ad5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:52.596033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.595999    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-certs" pod "eb5377b5-78d2-4523-b7cc-6eb386e15ad5" (UID: "eb5377b5-78d2-4523-b7cc-6eb386e15ad5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:52.596242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.596030    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:52.596242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.596055    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:52.596242 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.596117    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istiod-ca-cert" pod "eb5377b5-78d2-4523-b7cc-6eb386e15ad5" (UID: "eb5377b5-78d2-4523-b7cc-6eb386e15ad5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:32:52.597547 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.597518    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-envoy" pod "eb5377b5-78d2-4523-b7cc-6eb386e15ad5" (UID: "eb5377b5-78d2-4523-b7cc-6eb386e15ad5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:32:52.598165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.598138    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-podinfo" pod "eb5377b5-78d2-4523-b7cc-6eb386e15ad5" (UID: "eb5377b5-78d2-4523-b7cc-6eb386e15ad5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:32:52.598282 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.598205    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-token" pod "eb5377b5-78d2-4523-b7cc-6eb386e15ad5" (UID: "eb5377b5-78d2-4523-b7cc-6eb386e15ad5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:52.599050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.599024    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-kube-api-access-brl85" pod "eb5377b5-78d2-4523-b7cc-6eb386e15ad5" (UID: "eb5377b5-78d2-4523-b7cc-6eb386e15ad5"). InnerVolumeSpecName "kube-api-access-brl85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:32:52.653824 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.653788    4898 generic.go:356] "Generic (PLEG): container finished" podID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerID="bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5" exitCode=0
Nov 02 08:32:52.654065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.653833    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"
Nov 02 08:32:52.654065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.653866    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" event={"ID":"eb5377b5-78d2-4523-b7cc-6eb386e15ad5","Type":"ContainerDied","Data":"bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5"}
Nov 02 08:32:52.654065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.653958    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv" event={"ID":"eb5377b5-78d2-4523-b7cc-6eb386e15ad5","Type":"ContainerDied","Data":"ee8e9cc9b09a9be4a16d74bc91efec23170cf8373a3fe99560046316de1522cb"}
Nov 02 08:32:52.654065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.653992    4898 scope.go:122] "RemoveContainer" containerID="bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5"
Nov 02 08:32:52.669830 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.669800    4898 scope.go:122] "RemoveContainer" containerID="bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5"
Nov 02 08:32:52.670437 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:32:52.670409    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5\": container with ID starting with bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5 not found: ID does not exist" containerID="bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5"
Nov 02 08:32:52.670554 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.670449    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5"} err="failed to get container status \"bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5\": rpc error: code = NotFound desc = could not find container \"bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5\": container with ID starting with bb35a098a29bc4287371a3153733bd2762fd6b406ec2e63be65badbd70f5f4e5 not found: ID does not exist"
Nov 02 08:32:52.697812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.697780    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:52.697812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.697815    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brl85\" (UniqueName: \"kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-kube-api-access-brl85\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:52.698020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.697831    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:52.698020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.697848    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:52.698020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.697863    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:52.698020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.697880    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:52.698020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.697912    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb5377b5-78d2-4523-b7cc-6eb386e15ad5-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:32:52.708069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.708042    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"]
Nov 02 08:32:52.714797 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:52.714772    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7e7a83-openshift-default-6cdc8484df-dxchv"]
Nov 02 08:32:53.466755 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:32:53.466709    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" path="/var/lib/kubelet/pods/eb5377b5-78d2-4523-b7cc-6eb386e15ad5/volumes"
Nov 02 08:33:15.078155 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:33:15.078110    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:33:15.360334 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:15.360255    4898 scope.go:122] "RemoveContainer" containerID="e4c02f7cb4ee1430b96b723ddd7f53192ed148b2b23b57f48ed493265381d6e2"
Nov 02 08:33:44.989798 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:33:44.989758    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:33:58.599074 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.598977    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"]
Nov 02 08:33:58.599684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.599653    4898 cpu_manager.go:414] "containerMap: removing container" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerName="istio-proxy"
Nov 02 08:33:58.599684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.599672    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerName="istio-proxy"
Nov 02 08:33:58.599815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.599707    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerName="istio-proxy"
Nov 02 08:33:58.599815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.599716    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerName="istio-proxy"
Nov 02 08:33:58.599815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.599728    4898 cpu_manager.go:414] "containerMap: removing container" podUID="266b1f5a-899e-4175-9358-de87e3f7502a" containerName="payload-pre-processing"
Nov 02 08:33:58.599815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.599737    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266b1f5a-899e-4175-9358-de87e3f7502a" containerName="payload-pre-processing"
Nov 02 08:33:58.599815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.599759    4898 cpu_manager.go:414] "containerMap: removing container" podUID="35a88092-f516-49ce-9448-0f5a01ba4174" containerName="payload-processing"
Nov 02 08:33:58.599815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.599767    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a88092-f516-49ce-9448-0f5a01ba4174" containerName="payload-processing"
Nov 02 08:33:58.599815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.599793    4898 cpu_manager.go:414] "containerMap: removing container" podUID="29227432-f91a-4c80-9010-8c8c313a145c" containerName="revoke-keys"
Nov 02 08:33:58.599815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.599801    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29227432-f91a-4c80-9010-8c8c313a145c" containerName="revoke-keys"
Nov 02 08:33:58.600292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.599831    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerName="istio-proxy"
Nov 02 08:33:58.600292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.599836    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerName="istio-proxy"
Nov 02 08:33:58.600292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.600001    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerName="istio-proxy"
Nov 02 08:33:58.600292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.600016    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb5377b5-78d2-4523-b7cc-6eb386e15ad5" containerName="istio-proxy"
Nov 02 08:33:58.600292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.600034    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d217e2-96c8-4040-81d3-b24fdf9f5f2e" containerName="istio-proxy"
Nov 02 08:33:58.600292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.600043    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="266b1f5a-899e-4175-9358-de87e3f7502a" containerName="payload-pre-processing"
Nov 02 08:33:58.600292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.600055    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a88092-f516-49ce-9448-0f5a01ba4174" containerName="payload-processing"
Nov 02 08:33:58.600292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.600066    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="29227432-f91a-4c80-9010-8c8c313a145c" containerName="revoke-keys"
Nov 02 08:33:58.603606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.603579    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.631986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.631956    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"]
Nov 02 08:33:58.637603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.637579    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a44352\"/\"openshift-service-ca.crt\""
Nov 02 08:33:58.637792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.637662    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a44352\"/\"kube-root-ca.crt\""
Nov 02 08:33:58.637792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.637689    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a44352\"/\"rate-test-model-a44352-kserve-self-signed-certs\""
Nov 02 08:33:58.638012 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.637838    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a44352\"/\"default-dockercfg-jbqj2\""
Nov 02 08:33:58.679700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.679662    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tmp-dir\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.679839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.679733    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vcq\" (UniqueName: \"kubernetes.io/projected/ba9ac06a-7a08-4f47-85b6-89061ac53e66-kube-api-access-d6vcq\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.679929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.679879    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tls-certs\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.680005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.679991    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-dshm\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.680053 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.680020    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-home\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.680053 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.680049    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-model-cache\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.781016 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.780976    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-model-cache\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.781206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.781076    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tmp-dir\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.781206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.781133    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vcq\" (UniqueName: \"kubernetes.io/projected/ba9ac06a-7a08-4f47-85b6-89061ac53e66-kube-api-access-d6vcq\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.781341 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.781211    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tls-certs\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.781341 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.781277    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-dshm\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.781341 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.781329    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-home\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.781533 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.781436    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-model-cache\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.781595 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.781524    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tmp-dir\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.781974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.781946    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-home\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.783585 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.783561    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-dshm\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.783930 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.783907    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tls-certs\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.805368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.805339    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vcq\" (UniqueName: \"kubernetes.io/projected/ba9ac06a-7a08-4f47-85b6-89061ac53e66-kube-api-access-d6vcq\") pod \"rate-test-model-a44352-kserve-5df4d54b5-t2j9k\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") " pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:58.921915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:58.919138    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:33:59.260665 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:33:59.260633    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba9ac06a_7a08_4f47_85b6_89061ac53e66.slice/crio-3e55bb321411648b82fe185a40ef615ddfbd4a1f4aebd2d3f5844b089482e3da WatchSource:0}: Error finding container 3e55bb321411648b82fe185a40ef615ddfbd4a1f4aebd2d3f5844b089482e3da: Status 404 returned error can't find the container with id 3e55bb321411648b82fe185a40ef615ddfbd4a1f4aebd2d3f5844b089482e3da
Nov 02 08:33:59.260787 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:33:59.260661    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"]
Nov 02 08:34:00.202066 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:00.202025    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k" event={"ID":"ba9ac06a-7a08-4f47-85b6-89061ac53e66","Type":"ContainerStarted","Data":"faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7"}
Nov 02 08:34:00.202489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:00.202079    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k" event={"ID":"ba9ac06a-7a08-4f47-85b6-89061ac53e66","Type":"ContainerStarted","Data":"3e55bb321411648b82fe185a40ef615ddfbd4a1f4aebd2d3f5844b089482e3da"}
Nov 02 08:34:00.248501 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:00.248441    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k" podStartSLOduration=2.248421399 podStartE2EDuration="2.248421399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:34:00.242985072 +0000 UTC m=+4190.933312037" watchObservedRunningTime="2026-11-02 08:34:00.248421399 +0000 UTC m=+4190.938748369"
Nov 02 08:34:00.920807 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:00.920773    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:34:08.920345 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:08.920298    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:34:08.964656 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:08.964617    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:34:09.280481 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:09.280401    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:34:10.854802 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:10.854771    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:34:10.860568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:10.860538    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:34:14.991453 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:34:14.991409    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:34:15.408098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:15.408069    4898 scope.go:122] "RemoveContainer" containerID="5c052a387ad77177419701fd6f42f4d1ef63921e25a83366b52d0519875ae068"
Nov 02 08:34:31.058934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:31.058874    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"]
Nov 02 08:34:31.070352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:31.070326    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bada141-pbswd"]
Nov 02 08:34:31.467165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:31.467127    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2127178e-36c3-47c4-b407-c74773470512" path="/var/lib/kubelet/pods/2127178e-36c3-47c4-b407-c74773470512/volumes"
Nov 02 08:34:32.604528 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.604495    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"]
Nov 02 08:34:32.608553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.608535    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:32.627061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.627008    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:34:32.631196 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.631178    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"]
Nov 02 08:34:32.638968 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.638944    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:34:32.639096 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.639034    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:34:32.639167 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.639129    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:34:32.694504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.694467    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1258bc51-sthqm\" (UID: \"9105f772-fef3-420e-990b-f9a0658ca143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:32.694630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.694521    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd9sq\" (UniqueName: \"kubernetes.io/projected/9105f772-fef3-420e-990b-f9a0658ca143-kube-api-access-qd9sq\") pod \"maas-api-revoke-sub-1258bc51-sthqm\" (UID: \"9105f772-fef3-420e-990b-f9a0658ca143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:32.694680 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.694641    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1258bc51-sthqm\" (UID: \"9105f772-fef3-420e-990b-f9a0658ca143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:32.796309 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.796272    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd9sq\" (UniqueName: \"kubernetes.io/projected/9105f772-fef3-420e-990b-f9a0658ca143-kube-api-access-qd9sq\") pod \"maas-api-revoke-sub-1258bc51-sthqm\" (UID: \"9105f772-fef3-420e-990b-f9a0658ca143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:32.796477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.796426    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1258bc51-sthqm\" (UID: \"9105f772-fef3-420e-990b-f9a0658ca143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:32.796623 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.796605    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1258bc51-sthqm\" (UID: \"9105f772-fef3-420e-990b-f9a0658ca143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:32.797105 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.797082    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1258bc51-sthqm\" (UID: \"9105f772-fef3-420e-990b-f9a0658ca143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:32.798978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.798957    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1258bc51-sthqm\" (UID: \"9105f772-fef3-420e-990b-f9a0658ca143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:32.823100 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.823070    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd9sq\" (UniqueName: \"kubernetes.io/projected/9105f772-fef3-420e-990b-f9a0658ca143-kube-api-access-qd9sq\") pod \"maas-api-revoke-sub-1258bc51-sthqm\" (UID: \"9105f772-fef3-420e-990b-f9a0658ca143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:32.923317 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:32.923285    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:33.233324 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:33.233289    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"]
Nov 02 08:34:33.234616 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:34:33.234581    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9105f772_fef3_420e_990b_f9a0658ca143.slice/crio-8704e31765e81bc7d9c5bea2c88130cbcbe50c59c1c1aa4310daebf9b9842033 WatchSource:0}: Error finding container 8704e31765e81bc7d9c5bea2c88130cbcbe50c59c1c1aa4310daebf9b9842033: Status 404 returned error can't find the container with id 8704e31765e81bc7d9c5bea2c88130cbcbe50c59c1c1aa4310daebf9b9842033
Nov 02 08:34:33.499374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:33.499309    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm" event={"ID":"9105f772-fef3-420e-990b-f9a0658ca143","Type":"ContainerStarted","Data":"8704e31765e81bc7d9c5bea2c88130cbcbe50c59c1c1aa4310daebf9b9842033"}
Nov 02 08:34:34.510748 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:34.510654    4898 generic.go:356] "Generic (PLEG): container finished" podID="9105f772-fef3-420e-990b-f9a0658ca143" containerID="6038d72325187addc901b5e3177fb9dffc4d2010a5c26456d88091bf561ebc34" exitCode=0
Nov 02 08:34:34.510748 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:34.510735    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm" event={"ID":"9105f772-fef3-420e-990b-f9a0658ca143","Type":"ContainerDied","Data":"6038d72325187addc901b5e3177fb9dffc4d2010a5c26456d88091bf561ebc34"}
Nov 02 08:34:35.696535 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:35.696509    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:35.831798 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:35.831722    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-xks-ca\") pod \"9105f772-fef3-420e-990b-f9a0658ca143\" (UID: \"9105f772-fef3-420e-990b-f9a0658ca143\") "
Nov 02 08:34:35.831798 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:35.831761    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-openshift-ca\") pod \"9105f772-fef3-420e-990b-f9a0658ca143\" (UID: \"9105f772-fef3-420e-990b-f9a0658ca143\") "
Nov 02 08:34:35.832058 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:35.831874    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9105f772-fef3-420e-990b-f9a0658ca143-kube-api-access-qd9sq\" (UniqueName: \"kubernetes.io/projected/9105f772-fef3-420e-990b-f9a0658ca143-kube-api-access-qd9sq\") pod \"9105f772-fef3-420e-990b-f9a0658ca143\" (UID: \"9105f772-fef3-420e-990b-f9a0658ca143\") "
Nov 02 08:34:35.832297 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:35.832268    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-openshift-ca" pod "9105f772-fef3-420e-990b-f9a0658ca143" (UID: "9105f772-fef3-420e-990b-f9a0658ca143"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:34:35.834088 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:35.834062    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9105f772-fef3-420e-990b-f9a0658ca143-kube-api-access-qd9sq" pod "9105f772-fef3-420e-990b-f9a0658ca143" (UID: "9105f772-fef3-420e-990b-f9a0658ca143"). InnerVolumeSpecName "kube-api-access-qd9sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:34:35.834207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:35.834183    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-xks-ca" pod "9105f772-fef3-420e-990b-f9a0658ca143" (UID: "9105f772-fef3-420e-990b-f9a0658ca143"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:34:35.933806 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:35.933772    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd9sq\" (UniqueName: \"kubernetes.io/projected/9105f772-fef3-420e-990b-f9a0658ca143-kube-api-access-qd9sq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:34:35.933806 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:35.933803    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:34:35.934046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:35.933820    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9105f772-fef3-420e-990b-f9a0658ca143-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:34:36.526728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:36.526694    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm" event={"ID":"9105f772-fef3-420e-990b-f9a0658ca143","Type":"ContainerDied","Data":"8704e31765e81bc7d9c5bea2c88130cbcbe50c59c1c1aa4310daebf9b9842033"}
Nov 02 08:34:36.526932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:36.526749    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8704e31765e81bc7d9c5bea2c88130cbcbe50c59c1c1aa4310daebf9b9842033"
Nov 02 08:34:36.526932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:36.526705    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"
Nov 02 08:34:38.273325 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.273284    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"]
Nov 02 08:34:38.273923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.273906    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9105f772-fef3-420e-990b-f9a0658ca143" containerName="revoke-keys"
Nov 02 08:34:38.273923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.273923    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9105f772-fef3-420e-990b-f9a0658ca143" containerName="revoke-keys"
Nov 02 08:34:38.274049 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.274031    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9105f772-fef3-420e-990b-f9a0658ca143" containerName="revoke-keys"
Nov 02 08:34:38.277763 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.277741    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:38.301882 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.301849    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"]
Nov 02 08:34:38.308035 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.308006    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:34:38.308210 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.308158    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:34:38.309157 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.309134    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:34:38.309453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.309435    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:34:38.378770 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.378730    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc1b0c29-fg75p\" (UID: \"1f87f67d-575a-45df-8e5f-205e1eca5a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:38.378996 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.378798    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc1b0c29-fg75p\" (UID: \"1f87f67d-575a-45df-8e5f-205e1eca5a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:38.378996 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.378955    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8md\" (UniqueName: \"kubernetes.io/projected/1f87f67d-575a-45df-8e5f-205e1eca5a60-kube-api-access-6c8md\") pod \"maas-api-revoke-sub-fc1b0c29-fg75p\" (UID: \"1f87f67d-575a-45df-8e5f-205e1eca5a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:38.480812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.480772    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc1b0c29-fg75p\" (UID: \"1f87f67d-575a-45df-8e5f-205e1eca5a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:38.481046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.480840    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc1b0c29-fg75p\" (UID: \"1f87f67d-575a-45df-8e5f-205e1eca5a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:38.481046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.480982    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8md\" (UniqueName: \"kubernetes.io/projected/1f87f67d-575a-45df-8e5f-205e1eca5a60-kube-api-access-6c8md\") pod \"maas-api-revoke-sub-fc1b0c29-fg75p\" (UID: \"1f87f67d-575a-45df-8e5f-205e1eca5a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:38.481639 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.481606    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc1b0c29-fg75p\" (UID: \"1f87f67d-575a-45df-8e5f-205e1eca5a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:38.483572 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.483541    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc1b0c29-fg75p\" (UID: \"1f87f67d-575a-45df-8e5f-205e1eca5a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:38.504885 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.504855    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8md\" (UniqueName: \"kubernetes.io/projected/1f87f67d-575a-45df-8e5f-205e1eca5a60-kube-api-access-6c8md\") pod \"maas-api-revoke-sub-fc1b0c29-fg75p\" (UID: \"1f87f67d-575a-45df-8e5f-205e1eca5a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:38.592063 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.591968    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:38.907727 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:34:38.907693    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f87f67d_575a_45df_8e5f_205e1eca5a60.slice/crio-9235745a2a6cfe22f383d8d06e7f2b8a121689c17e10258f03eb0bd832397e7a WatchSource:0}: Error finding container 9235745a2a6cfe22f383d8d06e7f2b8a121689c17e10258f03eb0bd832397e7a: Status 404 returned error can't find the container with id 9235745a2a6cfe22f383d8d06e7f2b8a121689c17e10258f03eb0bd832397e7a
Nov 02 08:34:38.908991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:38.908967    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"]
Nov 02 08:34:39.549442 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:39.549365    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p" event={"ID":"1f87f67d-575a-45df-8e5f-205e1eca5a60","Type":"ContainerStarted","Data":"7567306e763f0bb941426d21b31a055fe856d715a3f44c1015cfa79413ae46bb"}
Nov 02 08:34:39.549442 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:39.549418    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p" event={"ID":"1f87f67d-575a-45df-8e5f-205e1eca5a60","Type":"ContainerStarted","Data":"9235745a2a6cfe22f383d8d06e7f2b8a121689c17e10258f03eb0bd832397e7a"}
Nov 02 08:34:39.607200 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:39.607135    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p" podStartSLOduration=1.6071167179999999 podStartE2EDuration="1.607116718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:34:39.592169235 +0000 UTC m=+4230.282496200" watchObservedRunningTime="2026-11-02 08:34:39.607116718 +0000 UTC m=+4230.297443683"
Nov 02 08:34:40.558942 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:40.558874    4898 generic.go:356] "Generic (PLEG): container finished" podID="1f87f67d-575a-45df-8e5f-205e1eca5a60" containerID="7567306e763f0bb941426d21b31a055fe856d715a3f44c1015cfa79413ae46bb" exitCode=0
Nov 02 08:34:40.559353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:40.558967    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p" event={"ID":"1f87f67d-575a-45df-8e5f-205e1eca5a60","Type":"ContainerDied","Data":"7567306e763f0bb941426d21b31a055fe856d715a3f44c1015cfa79413ae46bb"}
Nov 02 08:34:41.739590 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:41.739557    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:41.830940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:41.830786    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-xks-ca\") pod \"1f87f67d-575a-45df-8e5f-205e1eca5a60\" (UID: \"1f87f67d-575a-45df-8e5f-205e1eca5a60\") "
Nov 02 08:34:41.830940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:41.830863    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f87f67d-575a-45df-8e5f-205e1eca5a60-kube-api-access-6c8md\" (UniqueName: \"kubernetes.io/projected/1f87f67d-575a-45df-8e5f-205e1eca5a60-kube-api-access-6c8md\") pod \"1f87f67d-575a-45df-8e5f-205e1eca5a60\" (UID: \"1f87f67d-575a-45df-8e5f-205e1eca5a60\") "
Nov 02 08:34:41.831207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:41.831015    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-openshift-ca\") pod \"1f87f67d-575a-45df-8e5f-205e1eca5a60\" (UID: \"1f87f67d-575a-45df-8e5f-205e1eca5a60\") "
Nov 02 08:34:41.831429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:41.831395    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-openshift-ca" pod "1f87f67d-575a-45df-8e5f-205e1eca5a60" (UID: "1f87f67d-575a-45df-8e5f-205e1eca5a60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:34:41.831728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:41.831681    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:34:41.833496 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:41.833472    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-xks-ca" pod "1f87f67d-575a-45df-8e5f-205e1eca5a60" (UID: "1f87f67d-575a-45df-8e5f-205e1eca5a60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:34:41.836681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:41.836659    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f87f67d-575a-45df-8e5f-205e1eca5a60-kube-api-access-6c8md" pod "1f87f67d-575a-45df-8e5f-205e1eca5a60" (UID: "1f87f67d-575a-45df-8e5f-205e1eca5a60"). InnerVolumeSpecName "kube-api-access-6c8md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:34:41.932501 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:41.932465    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f87f67d-575a-45df-8e5f-205e1eca5a60-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:34:41.932501 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:41.932497    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c8md\" (UniqueName: \"kubernetes.io/projected/1f87f67d-575a-45df-8e5f-205e1eca5a60-kube-api-access-6c8md\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:34:42.576318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:42.576282    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"
Nov 02 08:34:42.576646 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:42.576310    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p" event={"ID":"1f87f67d-575a-45df-8e5f-205e1eca5a60","Type":"ContainerDied","Data":"9235745a2a6cfe22f383d8d06e7f2b8a121689c17e10258f03eb0bd832397e7a"}
Nov 02 08:34:42.576795 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:42.576669    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9235745a2a6cfe22f383d8d06e7f2b8a121689c17e10258f03eb0bd832397e7a"
Nov 02 08:34:43.104253 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.104212    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"]
Nov 02 08:34:43.104841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.104824    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1f87f67d-575a-45df-8e5f-205e1eca5a60" containerName="revoke-keys"
Nov 02 08:34:43.104924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.104843    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f87f67d-575a-45df-8e5f-205e1eca5a60" containerName="revoke-keys"
Nov 02 08:34:43.104986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.104975    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f87f67d-575a-45df-8e5f-205e1eca5a60" containerName="revoke-keys"
Nov 02 08:34:43.108570 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.108548    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:43.128546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.128517    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"]
Nov 02 08:34:43.136339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.136288    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:34:43.138566 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.138541    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:34:43.138711 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.138658    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:34:43.138791 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.138742    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:34:43.149423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.149398    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktxr9\" (UniqueName: \"kubernetes.io/projected/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-kube-api-access-ktxr9\") pod \"maas-api-revoke-sub-aa8b63b1-jj5md\" (UID: \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:43.149539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.149449    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa8b63b1-jj5md\" (UID: \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:43.149689 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.149667    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa8b63b1-jj5md\" (UID: \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:43.251407 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.251364    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktxr9\" (UniqueName: \"kubernetes.io/projected/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-kube-api-access-ktxr9\") pod \"maas-api-revoke-sub-aa8b63b1-jj5md\" (UID: \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:43.251592 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.251418    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa8b63b1-jj5md\" (UID: \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:43.251592 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.251558    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa8b63b1-jj5md\" (UID: \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:43.252293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.252264    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa8b63b1-jj5md\" (UID: \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:43.253957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.253934    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa8b63b1-jj5md\" (UID: \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:43.284782 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.284712    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktxr9\" (UniqueName: \"kubernetes.io/projected/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-kube-api-access-ktxr9\") pod \"maas-api-revoke-sub-aa8b63b1-jj5md\" (UID: \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:43.422990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.422958    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:43.743946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:43.743912    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"]
Nov 02 08:34:43.744767 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:34:43.744733    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9a22a6_ab15_473c_b6cc_0a00cc8ad371.slice/crio-a83cc0df629e92b684431ac908df8c27c4f4cf7c638a3a7e73cd1cd2c2f3c4dd WatchSource:0}: Error finding container a83cc0df629e92b684431ac908df8c27c4f4cf7c638a3a7e73cd1cd2c2f3c4dd: Status 404 returned error can't find the container with id a83cc0df629e92b684431ac908df8c27c4f4cf7c638a3a7e73cd1cd2c2f3c4dd
Nov 02 08:34:44.591975 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:44.591936    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md" event={"ID":"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371","Type":"ContainerStarted","Data":"cee1ce63261ce5f00dfd5c2dc033bf14beecda3deb7d760908489df5cc1b54c8"}
Nov 02 08:34:44.592446 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:44.591997    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md" event={"ID":"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371","Type":"ContainerStarted","Data":"a83cc0df629e92b684431ac908df8c27c4f4cf7c638a3a7e73cd1cd2c2f3c4dd"}
Nov 02 08:34:44.636489 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:44.636435    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md" podStartSLOduration=1.636420571 podStartE2EDuration="1.636420571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:34:44.63235586 +0000 UTC m=+4235.322682844" watchObservedRunningTime="2026-11-02 08:34:44.636420571 +0000 UTC m=+4235.326747587"
Nov 02 08:34:44.989956 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:34:44.989882    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:34:45.600705 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:45.600666    4898 generic.go:356] "Generic (PLEG): container finished" podID="fd9a22a6-ab15-473c-b6cc-0a00cc8ad371" containerID="cee1ce63261ce5f00dfd5c2dc033bf14beecda3deb7d760908489df5cc1b54c8" exitCode=0
Nov 02 08:34:45.601143 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:45.600722    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md" event={"ID":"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371","Type":"ContainerDied","Data":"cee1ce63261ce5f00dfd5c2dc033bf14beecda3deb7d760908489df5cc1b54c8"}
Nov 02 08:34:46.783517 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:46.783489    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:46.896987 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:46.896878    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-kube-api-access-ktxr9\" (UniqueName: \"kubernetes.io/projected/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-kube-api-access-ktxr9\") pod \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\" (UID: \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\") "
Nov 02 08:34:46.896987 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:46.896973    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-xks-ca\") pod \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\" (UID: \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\") "
Nov 02 08:34:46.897255 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:46.897022    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-openshift-ca\") pod \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\" (UID: \"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371\") "
Nov 02 08:34:46.897622 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:46.897584    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-openshift-ca" pod "fd9a22a6-ab15-473c-b6cc-0a00cc8ad371" (UID: "fd9a22a6-ab15-473c-b6cc-0a00cc8ad371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:34:46.899607 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:46.899563    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-kube-api-access-ktxr9" pod "fd9a22a6-ab15-473c-b6cc-0a00cc8ad371" (UID: "fd9a22a6-ab15-473c-b6cc-0a00cc8ad371"). InnerVolumeSpecName "kube-api-access-ktxr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:34:46.899945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:46.899876    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-xks-ca" pod "fd9a22a6-ab15-473c-b6cc-0a00cc8ad371" (UID: "fd9a22a6-ab15-473c-b6cc-0a00cc8ad371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:34:46.999191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:46.999163    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktxr9\" (UniqueName: \"kubernetes.io/projected/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-kube-api-access-ktxr9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:34:46.999191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:46.999189    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:34:46.999370 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:46.999199    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:34:47.617420 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:47.617382    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"
Nov 02 08:34:47.617613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:47.617413    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md" event={"ID":"fd9a22a6-ab15-473c-b6cc-0a00cc8ad371","Type":"ContainerDied","Data":"a83cc0df629e92b684431ac908df8c27c4f4cf7c638a3a7e73cd1cd2c2f3c4dd"}
Nov 02 08:34:47.617613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:47.617470    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a83cc0df629e92b684431ac908df8c27c4f4cf7c638a3a7e73cd1cd2c2f3c4dd"
Nov 02 08:34:48.591127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.591088    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"]
Nov 02 08:34:48.591672 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.591657    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fd9a22a6-ab15-473c-b6cc-0a00cc8ad371" containerName="revoke-keys"
Nov 02 08:34:48.591672 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.591673    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9a22a6-ab15-473c-b6cc-0a00cc8ad371" containerName="revoke-keys"
Nov 02 08:34:48.591804 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.591794    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9a22a6-ab15-473c-b6cc-0a00cc8ad371" containerName="revoke-keys"
Nov 02 08:34:48.595920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.595879    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:48.614060 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.614038    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"]
Nov 02 08:34:48.623175 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.623132    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:34:48.623321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.623218    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:34:48.645147 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.645120    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:34:48.645276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.645209    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:34:48.722052 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.722021    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25qtk\" (UniqueName: \"kubernetes.io/projected/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-kube-api-access-25qtk\") pod \"maas-api-revoke-sub-80122865-k95qx\" (UID: \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:48.722229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.722084    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80122865-k95qx\" (UID: \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:48.722229 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.722170    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80122865-k95qx\" (UID: \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:48.823703 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.823663    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25qtk\" (UniqueName: \"kubernetes.io/projected/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-kube-api-access-25qtk\") pod \"maas-api-revoke-sub-80122865-k95qx\" (UID: \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:48.823880 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.823735    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80122865-k95qx\" (UID: \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:48.823880 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.823759    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80122865-k95qx\" (UID: \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:48.824496 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.824467    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80122865-k95qx\" (UID: \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:48.826329 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.826295    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80122865-k95qx\" (UID: \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:48.846750 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.846694    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25qtk\" (UniqueName: \"kubernetes.io/projected/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-kube-api-access-25qtk\") pod \"maas-api-revoke-sub-80122865-k95qx\" (UID: \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:48.910916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:48.910849    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:49.226990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:49.226955    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"]
Nov 02 08:34:49.227914 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:34:49.227852    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7254db6_7bd2_4cea_8fa5_e2f7354fc9af.slice/crio-95b6d1ac03c26210433b7d0cfaf4b82bdb2e8f602ae9bf5047c1d609ffb15a48 WatchSource:0}: Error finding container 95b6d1ac03c26210433b7d0cfaf4b82bdb2e8f602ae9bf5047c1d609ffb15a48: Status 404 returned error can't find the container with id 95b6d1ac03c26210433b7d0cfaf4b82bdb2e8f602ae9bf5047c1d609ffb15a48
Nov 02 08:34:49.636920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:49.636851    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx" event={"ID":"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af","Type":"ContainerStarted","Data":"8d14175a9523a60a2a09467876b4b2e71a5393242151bcf22d3561775dc6ef7c"}
Nov 02 08:34:49.637286 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:49.636947    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx" event={"ID":"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af","Type":"ContainerStarted","Data":"95b6d1ac03c26210433b7d0cfaf4b82bdb2e8f602ae9bf5047c1d609ffb15a48"}
Nov 02 08:34:49.680809 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:49.680747    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx" podStartSLOduration=1.680727714 podStartE2EDuration="1.680727714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:34:49.6769623 +0000 UTC m=+4240.367289267" watchObservedRunningTime="2026-11-02 08:34:49.680727714 +0000 UTC m=+4240.371054684"
Nov 02 08:34:50.647946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:50.647837    4898 generic.go:356] "Generic (PLEG): container finished" podID="e7254db6-7bd2-4cea-8fa5-e2f7354fc9af" containerID="8d14175a9523a60a2a09467876b4b2e71a5393242151bcf22d3561775dc6ef7c" exitCode=0
Nov 02 08:34:50.648390 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:50.647940    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx" event={"ID":"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af","Type":"ContainerDied","Data":"8d14175a9523a60a2a09467876b4b2e71a5393242151bcf22d3561775dc6ef7c"}
Nov 02 08:34:51.820749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:51.820718    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:51.863843 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:51.863808    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-openshift-ca\") pod \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\" (UID: \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\") "
Nov 02 08:34:51.864015 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:51.863981    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-xks-ca\") pod \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\" (UID: \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\") "
Nov 02 08:34:51.864108 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:51.864091    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-kube-api-access-25qtk\" (UniqueName: \"kubernetes.io/projected/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-kube-api-access-25qtk\") pod \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\" (UID: \"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af\") "
Nov 02 08:34:51.864587 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:51.864557    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-openshift-ca" pod "e7254db6-7bd2-4cea-8fa5-e2f7354fc9af" (UID: "e7254db6-7bd2-4cea-8fa5-e2f7354fc9af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:34:51.867360 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:51.867318    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-xks-ca" pod "e7254db6-7bd2-4cea-8fa5-e2f7354fc9af" (UID: "e7254db6-7bd2-4cea-8fa5-e2f7354fc9af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:34:51.867501 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:51.867367    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-kube-api-access-25qtk" pod "e7254db6-7bd2-4cea-8fa5-e2f7354fc9af" (UID: "e7254db6-7bd2-4cea-8fa5-e2f7354fc9af"). InnerVolumeSpecName "kube-api-access-25qtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:34:51.966254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:51.966177    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:34:51.966254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:51.966206    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25qtk\" (UniqueName: \"kubernetes.io/projected/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-kube-api-access-25qtk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:34:51.966254 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:51.966217    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:34:52.665210 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:52.665180    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"
Nov 02 08:34:52.665432 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:52.665224    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx" event={"ID":"e7254db6-7bd2-4cea-8fa5-e2f7354fc9af","Type":"ContainerDied","Data":"95b6d1ac03c26210433b7d0cfaf4b82bdb2e8f602ae9bf5047c1d609ffb15a48"}
Nov 02 08:34:52.665432 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:52.665286    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95b6d1ac03c26210433b7d0cfaf4b82bdb2e8f602ae9bf5047c1d609ffb15a48"
Nov 02 08:34:59.511482 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.511444    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"]
Nov 02 08:34:59.512398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.512379    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e7254db6-7bd2-4cea-8fa5-e2f7354fc9af" containerName="revoke-keys"
Nov 02 08:34:59.512466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.512401    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7254db6-7bd2-4cea-8fa5-e2f7354fc9af" containerName="revoke-keys"
Nov 02 08:34:59.512578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.512566    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7254db6-7bd2-4cea-8fa5-e2f7354fc9af" containerName="revoke-keys"
Nov 02 08:34:59.516759 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.516736    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:34:59.531591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.531565    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:34:59.534310 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.534283    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"]
Nov 02 08:34:59.539366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.539343    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:34:59.543813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.543787    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:34:59.543937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.543864    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:34:59.693048 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.693015    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj5mk\" (UniqueName: \"kubernetes.io/projected/f0127b61-caef-4959-b4ad-f34e72e25bde-kube-api-access-vj5mk\") pod \"maas-api-revoke-sub-46aca450-m4gp7\" (UID: \"f0127b61-caef-4959-b4ad-f34e72e25bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:34:59.693245 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.693186    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46aca450-m4gp7\" (UID: \"f0127b61-caef-4959-b4ad-f34e72e25bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:34:59.693312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.693248    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46aca450-m4gp7\" (UID: \"f0127b61-caef-4959-b4ad-f34e72e25bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:34:59.794217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.794135    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46aca450-m4gp7\" (UID: \"f0127b61-caef-4959-b4ad-f34e72e25bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:34:59.794391 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.794281    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj5mk\" (UniqueName: \"kubernetes.io/projected/f0127b61-caef-4959-b4ad-f34e72e25bde-kube-api-access-vj5mk\") pod \"maas-api-revoke-sub-46aca450-m4gp7\" (UID: \"f0127b61-caef-4959-b4ad-f34e72e25bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:34:59.794444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.794398    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46aca450-m4gp7\" (UID: \"f0127b61-caef-4959-b4ad-f34e72e25bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:34:59.795076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.795054    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46aca450-m4gp7\" (UID: \"f0127b61-caef-4959-b4ad-f34e72e25bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:34:59.796709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.796677    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46aca450-m4gp7\" (UID: \"f0127b61-caef-4959-b4ad-f34e72e25bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:34:59.817667 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.817641    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj5mk\" (UniqueName: \"kubernetes.io/projected/f0127b61-caef-4959-b4ad-f34e72e25bde-kube-api-access-vj5mk\") pod \"maas-api-revoke-sub-46aca450-m4gp7\" (UID: \"f0127b61-caef-4959-b4ad-f34e72e25bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:34:59.852907 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:34:59.852098    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:35:00.163225 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:35:00.163178    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0127b61_caef_4959_b4ad_f34e72e25bde.slice/crio-34e4dfe69de2d999fdd3db3076032cac02c843d45feabfb818fb045bf77f968d WatchSource:0}: Error finding container 34e4dfe69de2d999fdd3db3076032cac02c843d45feabfb818fb045bf77f968d: Status 404 returned error can't find the container with id 34e4dfe69de2d999fdd3db3076032cac02c843d45feabfb818fb045bf77f968d
Nov 02 08:35:00.163346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:00.163320    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"]
Nov 02 08:35:00.731990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:00.731876    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7" event={"ID":"f0127b61-caef-4959-b4ad-f34e72e25bde","Type":"ContainerStarted","Data":"cb079292079498cca01cd4ceb6d11479446622eaac2af0145a10846b9d58a4e4"}
Nov 02 08:35:00.731990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:00.731953    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7" event={"ID":"f0127b61-caef-4959-b4ad-f34e72e25bde","Type":"ContainerStarted","Data":"34e4dfe69de2d999fdd3db3076032cac02c843d45feabfb818fb045bf77f968d"}
Nov 02 08:35:00.774099 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:00.774048    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7" podStartSLOduration=1.7740339280000001 podStartE2EDuration="1.774033928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:35:00.771968241 +0000 UTC m=+4251.462295240" watchObservedRunningTime="2026-11-02 08:35:00.774033928 +0000 UTC m=+4251.464360892"
Nov 02 08:35:01.741932 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:01.741871    4898 generic.go:356] "Generic (PLEG): container finished" podID="f0127b61-caef-4959-b4ad-f34e72e25bde" containerID="cb079292079498cca01cd4ceb6d11479446622eaac2af0145a10846b9d58a4e4" exitCode=0
Nov 02 08:35:01.742323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:01.741937    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7" event={"ID":"f0127b61-caef-4959-b4ad-f34e72e25bde","Type":"ContainerDied","Data":"cb079292079498cca01cd4ceb6d11479446622eaac2af0145a10846b9d58a4e4"}
Nov 02 08:35:02.936291 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:02.936258    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:35:03.051008 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:03.050911    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-openshift-ca\") pod \"f0127b61-caef-4959-b4ad-f34e72e25bde\" (UID: \"f0127b61-caef-4959-b4ad-f34e72e25bde\") "
Nov 02 08:35:03.051008 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:03.050980    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0127b61-caef-4959-b4ad-f34e72e25bde-kube-api-access-vj5mk\" (UniqueName: \"kubernetes.io/projected/f0127b61-caef-4959-b4ad-f34e72e25bde-kube-api-access-vj5mk\") pod \"f0127b61-caef-4959-b4ad-f34e72e25bde\" (UID: \"f0127b61-caef-4959-b4ad-f34e72e25bde\") "
Nov 02 08:35:03.051260 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:03.051133    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-xks-ca\") pod \"f0127b61-caef-4959-b4ad-f34e72e25bde\" (UID: \"f0127b61-caef-4959-b4ad-f34e72e25bde\") "
Nov 02 08:35:03.051260 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:03.051222    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-openshift-ca" pod "f0127b61-caef-4959-b4ad-f34e72e25bde" (UID: "f0127b61-caef-4959-b4ad-f34e72e25bde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:35:03.051658 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:03.051638    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:03.053190 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:03.053156    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0127b61-caef-4959-b4ad-f34e72e25bde-kube-api-access-vj5mk" pod "f0127b61-caef-4959-b4ad-f34e72e25bde" (UID: "f0127b61-caef-4959-b4ad-f34e72e25bde"). InnerVolumeSpecName "kube-api-access-vj5mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:03.053306 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:03.053292    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-xks-ca" pod "f0127b61-caef-4959-b4ad-f34e72e25bde" (UID: "f0127b61-caef-4959-b4ad-f34e72e25bde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:35:03.152589 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:03.152549    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj5mk\" (UniqueName: \"kubernetes.io/projected/f0127b61-caef-4959-b4ad-f34e72e25bde-kube-api-access-vj5mk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:03.152589 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:03.152583    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0127b61-caef-4959-b4ad-f34e72e25bde-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:03.773813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:03.772760    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7" event={"ID":"f0127b61-caef-4959-b4ad-f34e72e25bde","Type":"ContainerDied","Data":"34e4dfe69de2d999fdd3db3076032cac02c843d45feabfb818fb045bf77f968d"}
Nov 02 08:35:03.773813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:03.772830    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34e4dfe69de2d999fdd3db3076032cac02c843d45feabfb818fb045bf77f968d"
Nov 02 08:35:03.773813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:03.772915    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"
Nov 02 08:35:10.256916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.256511    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"]
Nov 02 08:35:10.257530 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.257473    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f0127b61-caef-4959-b4ad-f34e72e25bde" containerName="revoke-keys"
Nov 02 08:35:10.257530 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.257501    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0127b61-caef-4959-b4ad-f34e72e25bde" containerName="revoke-keys"
Nov 02 08:35:10.257683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.257658    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0127b61-caef-4959-b4ad-f34e72e25bde" containerName="revoke-keys"
Nov 02 08:35:10.269764 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.269732    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:10.290203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.290165    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:35:10.305129 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.304387    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:35:10.305129 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.304512    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:35:10.305129 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.304611    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:35:10.343513 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.342254    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"]
Nov 02 08:35:10.398649 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.398613    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d9e33ca-sdhzs\" (UID: \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:10.398823 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.398736    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d9e33ca-sdhzs\" (UID: \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:10.398927 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.398882    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl2k8\" (UniqueName: \"kubernetes.io/projected/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-kube-api-access-tl2k8\") pod \"maas-api-revoke-sub-2d9e33ca-sdhzs\" (UID: \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:10.500630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.500591    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d9e33ca-sdhzs\" (UID: \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:10.500817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.500718    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl2k8\" (UniqueName: \"kubernetes.io/projected/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-kube-api-access-tl2k8\") pod \"maas-api-revoke-sub-2d9e33ca-sdhzs\" (UID: \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:10.500817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.500798    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d9e33ca-sdhzs\" (UID: \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:10.501456 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.501426    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d9e33ca-sdhzs\" (UID: \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:10.503117 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.503089    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d9e33ca-sdhzs\" (UID: \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:10.524399 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.524341    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl2k8\" (UniqueName: \"kubernetes.io/projected/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-kube-api-access-tl2k8\") pod \"maas-api-revoke-sub-2d9e33ca-sdhzs\" (UID: \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:10.619835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.619801    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:10.723568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.723534    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"]
Nov 02 08:35:10.966538 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:10.966495    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"]
Nov 02 08:35:10.973228 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:35:10.973193    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e98831c_cc04_405f_a7fb_d8c19bf60bcb.slice/crio-1e94be104e894b07edb566f6eb21c5e68094b0861b6df3d69522d6ee6a1dd433 WatchSource:0}: Error finding container 1e94be104e894b07edb566f6eb21c5e68094b0861b6df3d69522d6ee6a1dd433: Status 404 returned error can't find the container with id 1e94be104e894b07edb566f6eb21c5e68094b0861b6df3d69522d6ee6a1dd433
Nov 02 08:35:11.857297 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:11.857253    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs" event={"ID":"0e98831c-cc04-405f-a7fb-d8c19bf60bcb","Type":"ContainerStarted","Data":"79520f955e496390946406a90a507e901c91d57dc40dd0fc89097de0b0b2eec0"}
Nov 02 08:35:11.857815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:11.857324    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs" event={"ID":"0e98831c-cc04-405f-a7fb-d8c19bf60bcb","Type":"ContainerStarted","Data":"1e94be104e894b07edb566f6eb21c5e68094b0861b6df3d69522d6ee6a1dd433"}
Nov 02 08:35:11.899076 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:11.899028    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs" podStartSLOduration=1.8990134969999999 podStartE2EDuration="1.899013497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:35:11.896839241 +0000 UTC m=+4262.587166230" watchObservedRunningTime="2026-11-02 08:35:11.899013497 +0000 UTC m=+4262.589340460"
Nov 02 08:35:12.867911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:12.867857    4898 generic.go:356] "Generic (PLEG): container finished" podID="0e98831c-cc04-405f-a7fb-d8c19bf60bcb" containerID="79520f955e496390946406a90a507e901c91d57dc40dd0fc89097de0b0b2eec0" exitCode=0
Nov 02 08:35:12.868361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:12.867933    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs" event={"ID":"0e98831c-cc04-405f-a7fb-d8c19bf60bcb","Type":"ContainerDied","Data":"79520f955e496390946406a90a507e901c91d57dc40dd0fc89097de0b0b2eec0"}
Nov 02 08:35:14.027179 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.027146    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:14.158608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.158569    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-xks-ca\") pod \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\" (UID: \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\") "
Nov 02 08:35:14.158783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.158644    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-openshift-ca\") pod \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\" (UID: \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\") "
Nov 02 08:35:14.158850 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.158835    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-kube-api-access-tl2k8\" (UniqueName: \"kubernetes.io/projected/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-kube-api-access-tl2k8\") pod \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\" (UID: \"0e98831c-cc04-405f-a7fb-d8c19bf60bcb\") "
Nov 02 08:35:14.159060 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.159026    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-openshift-ca" pod "0e98831c-cc04-405f-a7fb-d8c19bf60bcb" (UID: "0e98831c-cc04-405f-a7fb-d8c19bf60bcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:35:14.159397 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.159376    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:14.160976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.160941    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-xks-ca" pod "0e98831c-cc04-405f-a7fb-d8c19bf60bcb" (UID: "0e98831c-cc04-405f-a7fb-d8c19bf60bcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:35:14.162029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.162006    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-kube-api-access-tl2k8" pod "0e98831c-cc04-405f-a7fb-d8c19bf60bcb" (UID: "0e98831c-cc04-405f-a7fb-d8c19bf60bcb"). InnerVolumeSpecName "kube-api-access-tl2k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:14.260744 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.260706    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl2k8\" (UniqueName: \"kubernetes.io/projected/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-kube-api-access-tl2k8\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:14.260744 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.260745    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e98831c-cc04-405f-a7fb-d8c19bf60bcb-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:14.886573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.886537    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"
Nov 02 08:35:14.886573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.886558    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs" event={"ID":"0e98831c-cc04-405f-a7fb-d8c19bf60bcb","Type":"ContainerDied","Data":"1e94be104e894b07edb566f6eb21c5e68094b0861b6df3d69522d6ee6a1dd433"}
Nov 02 08:35:14.886810 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:14.886620    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e94be104e894b07edb566f6eb21c5e68094b0861b6df3d69522d6ee6a1dd433"
Nov 02 08:35:14.991512 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:35:14.991471    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:35:15.724107 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:15.723961    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f" podUID="4e7cea32-8481-4054-89c1-6b72c41ea4f7" containerName="payload-pre-processing" containerID="cri-o://a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce" gracePeriod=25
Nov 02 08:35:19.095427 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:19.095376    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"]
Nov 02 08:35:19.096248 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:19.096212    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy" containerID="cri-o://514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a" gracePeriod=30
Nov 02 08:35:21.031716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:21.031670    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f" podUID="4e7cea32-8481-4054-89c1-6b72c41ea4f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:35:22.559815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:22.559734    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:35:22.559815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:22.559802    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:35:24.383951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.383916    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 08:35:24.435188 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.435155    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9609fcde-fbb5-409b-82bf-78eaec870453-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9609fcde-fbb5-409b-82bf-78eaec870453-istio-podinfo\") pod \"9609fcde-fbb5-409b-82bf-78eaec870453\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") "
Nov 02 08:35:24.435410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.435276    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9609fcde-fbb5-409b-82bf-78eaec870453-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9609fcde-fbb5-409b-82bf-78eaec870453-istiod-ca-cert\") pod \"9609fcde-fbb5-409b-82bf-78eaec870453\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") "
Nov 02 08:35:24.435410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.435313    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-envoy\") pod \"9609fcde-fbb5-409b-82bf-78eaec870453\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") "
Nov 02 08:35:24.435614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.435581    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-istio-token\" (UniqueName: \"kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-istio-token\") pod \"9609fcde-fbb5-409b-82bf-78eaec870453\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") "
Nov 02 08:35:24.435751 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.435648    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-certs\") pod \"9609fcde-fbb5-409b-82bf-78eaec870453\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") "
Nov 02 08:35:24.435751 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.435731    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-data\") pod \"9609fcde-fbb5-409b-82bf-78eaec870453\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") "
Nov 02 08:35:24.435751 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.435732    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9609fcde-fbb5-409b-82bf-78eaec870453-istiod-ca-cert" pod "9609fcde-fbb5-409b-82bf-78eaec870453" (UID: "9609fcde-fbb5-409b-82bf-78eaec870453"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:35:24.435978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.435779    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-kube-api-access-2nn7l\" (UniqueName: \"kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-kube-api-access-2nn7l\") pod \"9609fcde-fbb5-409b-82bf-78eaec870453\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") "
Nov 02 08:35:24.435978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.435812    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-credential-socket\") pod \"9609fcde-fbb5-409b-82bf-78eaec870453\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") "
Nov 02 08:35:24.435978 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.435910    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-socket\") pod \"9609fcde-fbb5-409b-82bf-78eaec870453\" (UID: \"9609fcde-fbb5-409b-82bf-78eaec870453\") "
Nov 02 08:35:24.436156 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.436129    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-data" pod "9609fcde-fbb5-409b-82bf-78eaec870453" (UID: "9609fcde-fbb5-409b-82bf-78eaec870453"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:24.436516 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.436372    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-socket" pod "9609fcde-fbb5-409b-82bf-78eaec870453" (UID: "9609fcde-fbb5-409b-82bf-78eaec870453"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:24.436516 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.436457    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-certs" pod "9609fcde-fbb5-409b-82bf-78eaec870453" (UID: "9609fcde-fbb5-409b-82bf-78eaec870453"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:24.436516 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.436482    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-credential-socket" pod "9609fcde-fbb5-409b-82bf-78eaec870453" (UID: "9609fcde-fbb5-409b-82bf-78eaec870453"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:24.436728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.436567    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:24.436728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.436586    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:24.436728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.436599    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9609fcde-fbb5-409b-82bf-78eaec870453-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:24.438350 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.438303    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-envoy" pod "9609fcde-fbb5-409b-82bf-78eaec870453" (UID: "9609fcde-fbb5-409b-82bf-78eaec870453"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:35:24.438429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.438406    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9609fcde-fbb5-409b-82bf-78eaec870453-istio-podinfo" pod "9609fcde-fbb5-409b-82bf-78eaec870453" (UID: "9609fcde-fbb5-409b-82bf-78eaec870453"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:35:24.438555 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.438537    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-kube-api-access-2nn7l" pod "9609fcde-fbb5-409b-82bf-78eaec870453" (UID: "9609fcde-fbb5-409b-82bf-78eaec870453"). InnerVolumeSpecName "kube-api-access-2nn7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:24.438604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.438566    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-istio-token" pod "9609fcde-fbb5-409b-82bf-78eaec870453" (UID: "9609fcde-fbb5-409b-82bf-78eaec870453"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:24.537824 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.537783    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nn7l\" (UniqueName: \"kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-kube-api-access-2nn7l\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:24.537824 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.537820    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:24.538099 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.537836    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9609fcde-fbb5-409b-82bf-78eaec870453-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:24.538099 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.537849    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:24.538099 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.537862    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9609fcde-fbb5-409b-82bf-78eaec870453-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:24.538099 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.537874    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9609fcde-fbb5-409b-82bf-78eaec870453-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:24.983742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.983709    4898 generic.go:356] "Generic (PLEG): container finished" podID="9609fcde-fbb5-409b-82bf-78eaec870453" containerID="514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a" exitCode=0
Nov 02 08:35:24.983951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.983785    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"
Nov 02 08:35:24.983951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.983800    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" event={"ID":"9609fcde-fbb5-409b-82bf-78eaec870453","Type":"ContainerDied","Data":"514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a"}
Nov 02 08:35:24.983951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.983869    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j" event={"ID":"9609fcde-fbb5-409b-82bf-78eaec870453","Type":"ContainerDied","Data":"9901175637f9914a9eaabaefcb1ccb7f2d3257799474dbcd35f1729aff82cfa6"}
Nov 02 08:35:24.983951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.983927    4898 scope.go:122] "RemoveContainer" containerID="514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a"
Nov 02 08:35:24.995919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:24.995882    4898 scope.go:122] "RemoveContainer" containerID="9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e"
Nov 02 08:35:25.006807 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:25.006787    4898 scope.go:122] "RemoveContainer" containerID="514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a"
Nov 02 08:35:25.007117 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:35:25.007096    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a\": container with ID starting with 514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a not found: ID does not exist" containerID="514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a"
Nov 02 08:35:25.007195 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:25.007126    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a"} err="failed to get container status \"514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a\": rpc error: code = NotFound desc = could not find container \"514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a\": container with ID starting with 514daa925eeec6ac4f29e0e47bcba9ca37744fcdc85d07e7572411e68177d99a not found: ID does not exist"
Nov 02 08:35:25.007195 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:25.007146    4898 scope.go:122] "RemoveContainer" containerID="9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e"
Nov 02 08:35:25.007364 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:35:25.007348    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e\": container with ID starting with 9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e not found: ID does not exist" containerID="9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e"
Nov 02 08:35:25.007405 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:25.007368    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e"} err="failed to get container status \"9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e\": rpc error: code = NotFound desc = could not find container \"9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e\": container with ID starting with 9d45748543325eb8ab30caccaf5ee8879fc55652c61c05a84b46f552f685a20e not found: ID does not exist"
Nov 02 08:35:25.052420 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:25.052393    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"]
Nov 02 08:35:25.064795 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:25.064763    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-696c7a-openshift-default-67695fc869-j578j"]
Nov 02 08:35:25.466859 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:25.466815    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" path="/var/lib/kubelet/pods/9609fcde-fbb5-409b-82bf-78eaec870453/volumes"
Nov 02 08:35:31.038299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:31.038251    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f" podUID="4e7cea32-8481-4054-89c1-6b72c41ea4f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:35:35.978310 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:35.978278    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"
Nov 02 08:35:36.012652 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.012622    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7cea32-8481-4054-89c1-6b72c41ea4f7-kube-api-access-9dksp\" (UniqueName: \"kubernetes.io/projected/4e7cea32-8481-4054-89c1-6b72c41ea4f7-kube-api-access-9dksp\") pod \"4e7cea32-8481-4054-89c1-6b72c41ea4f7\" (UID: \"4e7cea32-8481-4054-89c1-6b72c41ea4f7\") "
Nov 02 08:35:36.012816 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.012719    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7cea32-8481-4054-89c1-6b72c41ea4f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7cea32-8481-4054-89c1-6b72c41ea4f7-plugins-config-volume\") pod \"4e7cea32-8481-4054-89c1-6b72c41ea4f7\" (UID: \"4e7cea32-8481-4054-89c1-6b72c41ea4f7\") "
Nov 02 08:35:36.013186 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.013163    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7cea32-8481-4054-89c1-6b72c41ea4f7-plugins-config-volume" pod "4e7cea32-8481-4054-89c1-6b72c41ea4f7" (UID: "4e7cea32-8481-4054-89c1-6b72c41ea4f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:35:36.014808 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.014777    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7cea32-8481-4054-89c1-6b72c41ea4f7-kube-api-access-9dksp" pod "4e7cea32-8481-4054-89c1-6b72c41ea4f7" (UID: "4e7cea32-8481-4054-89c1-6b72c41ea4f7"). InnerVolumeSpecName "kube-api-access-9dksp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:35:36.103363 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.103272    4898 generic.go:356] "Generic (PLEG): container finished" podID="4e7cea32-8481-4054-89c1-6b72c41ea4f7" containerID="a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce" exitCode=0
Nov 02 08:35:36.103363 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.103336    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"
Nov 02 08:35:36.103363 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.103347    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f" event={"ID":"4e7cea32-8481-4054-89c1-6b72c41ea4f7","Type":"ContainerDied","Data":"a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce"}
Nov 02 08:35:36.103613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.103414    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f" event={"ID":"4e7cea32-8481-4054-89c1-6b72c41ea4f7","Type":"ContainerDied","Data":"336299f82b54b61f06e76bb95977a41bf4fc7b95ad67b9591156f04816c49c88"}
Nov 02 08:35:36.103613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.103438    4898 scope.go:122] "RemoveContainer" containerID="a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce"
Nov 02 08:35:36.113946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.113918    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dksp\" (UniqueName: \"kubernetes.io/projected/4e7cea32-8481-4054-89c1-6b72c41ea4f7-kube-api-access-9dksp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:36.114056 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.113949    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7cea32-8481-4054-89c1-6b72c41ea4f7-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:35:36.115374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.115350    4898 scope.go:122] "RemoveContainer" containerID="a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce"
Nov 02 08:35:36.115640 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:35:36.115621    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce\": container with ID starting with a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce not found: ID does not exist" containerID="a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce"
Nov 02 08:35:36.115704 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.115649    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce"} err="failed to get container status \"a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce\": rpc error: code = NotFound desc = could not find container \"a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce\": container with ID starting with a916a8ac8b116877309d377dc6e03d6f59c5554d1c824ef809f983172c4bf1ce not found: ID does not exist"
Nov 02 08:35:36.199973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.199935    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"]
Nov 02 08:35:36.209375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:36.209310    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-696c7a-d6965f547-wh87f"]
Nov 02 08:35:37.479330 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:37.479297    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7cea32-8481-4054-89c1-6b72c41ea4f7" path="/var/lib/kubelet/pods/4e7cea32-8481-4054-89c1-6b72c41ea4f7/volumes"
Nov 02 08:35:44.992477 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:35:44.992439    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:35:58.513079 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:58.513038    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"]
Nov 02 08:35:58.513584 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:35:58.513320    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy" containerID="cri-o://d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69" gracePeriod=30
Nov 02 08:36:02.079185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:02.079144    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"]
Nov 02 08:36:02.079826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:02.079774    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy" containerID="cri-o://21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd" gracePeriod=30
Nov 02 08:36:02.224934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:02.221801    4898 patch_prober.go:29] interesting pod/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:36:02.224934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:02.221877    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:36:03.810613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.810584    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:36:03.939959 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.939926    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-envoy\") pod \"3341400d-3694-48ec-86e8-e0ed90f5776b\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") "
Nov 02 08:36:03.940144 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.939983    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-token\" (UniqueName: \"kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-token\") pod \"3341400d-3694-48ec-86e8-e0ed90f5776b\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") "
Nov 02 08:36:03.940144 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.940043    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-data\") pod \"3341400d-3694-48ec-86e8-e0ed90f5776b\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") "
Nov 02 08:36:03.940144 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.940067    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-credential-socket\") pod \"3341400d-3694-48ec-86e8-e0ed90f5776b\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") "
Nov 02 08:36:03.940144 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.940127    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-certs\") pod \"3341400d-3694-48ec-86e8-e0ed90f5776b\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") "
Nov 02 08:36:03.940381 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.940194    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-podinfo\") pod \"3341400d-3694-48ec-86e8-e0ed90f5776b\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") "
Nov 02 08:36:03.940381 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.940287    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-socket\") pod \"3341400d-3694-48ec-86e8-e0ed90f5776b\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") "
Nov 02 08:36:03.940381 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.940338    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3341400d-3694-48ec-86e8-e0ed90f5776b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3341400d-3694-48ec-86e8-e0ed90f5776b-istiod-ca-cert\") pod \"3341400d-3694-48ec-86e8-e0ed90f5776b\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") "
Nov 02 08:36:03.940537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.940371    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-data" pod "3341400d-3694-48ec-86e8-e0ed90f5776b" (UID: "3341400d-3694-48ec-86e8-e0ed90f5776b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:03.940537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.940378    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-kube-api-access-sbmwp\" (UniqueName: \"kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-kube-api-access-sbmwp\") pod \"3341400d-3694-48ec-86e8-e0ed90f5776b\" (UID: \"3341400d-3694-48ec-86e8-e0ed90f5776b\") "
Nov 02 08:36:03.940537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.940450    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-credential-socket" pod "3341400d-3694-48ec-86e8-e0ed90f5776b" (UID: "3341400d-3694-48ec-86e8-e0ed90f5776b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:03.940691 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.940595    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-socket" pod "3341400d-3694-48ec-86e8-e0ed90f5776b" (UID: "3341400d-3694-48ec-86e8-e0ed90f5776b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:03.940815 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.940791    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3341400d-3694-48ec-86e8-e0ed90f5776b-istiod-ca-cert" pod "3341400d-3694-48ec-86e8-e0ed90f5776b" (UID: "3341400d-3694-48ec-86e8-e0ed90f5776b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:03.940967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.940944    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-certs" pod "3341400d-3694-48ec-86e8-e0ed90f5776b" (UID: "3341400d-3694-48ec-86e8-e0ed90f5776b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:03.941066 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.941046    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:03.941128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.941070    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:03.941128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.941085    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:03.941128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.941102    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:03.941128 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.941115    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3341400d-3694-48ec-86e8-e0ed90f5776b-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:03.942819 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.942793    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-envoy" pod "3341400d-3694-48ec-86e8-e0ed90f5776b" (UID: "3341400d-3694-48ec-86e8-e0ed90f5776b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:03.942954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.942834    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-token" pod "3341400d-3694-48ec-86e8-e0ed90f5776b" (UID: "3341400d-3694-48ec-86e8-e0ed90f5776b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:03.943024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.942994    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-podinfo" pod "3341400d-3694-48ec-86e8-e0ed90f5776b" (UID: "3341400d-3694-48ec-86e8-e0ed90f5776b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:36:03.943148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:03.943019    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-kube-api-access-sbmwp" pod "3341400d-3694-48ec-86e8-e0ed90f5776b" (UID: "3341400d-3694-48ec-86e8-e0ed90f5776b"). InnerVolumeSpecName "kube-api-access-sbmwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:04.042870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.042798    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbmwp\" (UniqueName: \"kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-kube-api-access-sbmwp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:04.042870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.042827    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:04.042870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.042839    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:04.042870 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.042847    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3341400d-3694-48ec-86e8-e0ed90f5776b-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:04.380080 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.379987    4898 generic.go:356] "Generic (PLEG): container finished" podID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerID="d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69" exitCode=0
Nov 02 08:36:04.380080 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.380040    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" event={"ID":"3341400d-3694-48ec-86e8-e0ed90f5776b","Type":"ContainerDied","Data":"d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69"}
Nov 02 08:36:04.380318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.380090    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz" event={"ID":"3341400d-3694-48ec-86e8-e0ed90f5776b","Type":"ContainerDied","Data":"7b0216bac6f83ecd0ecbc7c7abb62065a3aa80f3ec6f364ae5c3f98c48b163d1"}
Nov 02 08:36:04.380318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.380046    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"
Nov 02 08:36:04.380318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.380118    4898 scope.go:122] "RemoveContainer" containerID="d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69"
Nov 02 08:36:04.405144 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.396040    4898 scope.go:122] "RemoveContainer" containerID="cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00"
Nov 02 08:36:04.411123 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.411086    4898 scope.go:122] "RemoveContainer" containerID="d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69"
Nov 02 08:36:04.411453 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:36:04.411425    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69\": container with ID starting with d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69 not found: ID does not exist" containerID="d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69"
Nov 02 08:36:04.411564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.411459    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69"} err="failed to get container status \"d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69\": rpc error: code = NotFound desc = could not find container \"d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69\": container with ID starting with d64b066a4058615042775ddd172a9addb1d933f285ab1bca8b34f86f827a1c69 not found: ID does not exist"
Nov 02 08:36:04.411564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.411478    4898 scope.go:122] "RemoveContainer" containerID="cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00"
Nov 02 08:36:04.411837 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:36:04.411813    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00\": container with ID starting with cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00 not found: ID does not exist" containerID="cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00"
Nov 02 08:36:04.411916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.411847    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00"} err="failed to get container status \"cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00\": rpc error: code = NotFound desc = could not find container \"cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00\": container with ID starting with cc72c64fda2b401c1ffd23a23036a20edab9c75150bd48a018b79f9252e1bd00 not found: ID does not exist"
Nov 02 08:36:04.441914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.435610    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"]
Nov 02 08:36:04.454004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:04.453971    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-643355-openshift-default-77b8b64ffc-wddxz"]
Nov 02 08:36:05.481585 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:05.480097    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" path="/var/lib/kubelet/pods/3341400d-3694-48ec-86e8-e0ed90f5776b/volumes"
Nov 02 08:36:06.636802 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.636765    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"]
Nov 02 08:36:06.637368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637350    4898 cpu_manager.go:414] "containerMap: removing container" podUID="4e7cea32-8481-4054-89c1-6b72c41ea4f7" containerName="payload-pre-processing"
Nov 02 08:36:06.637368 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637368    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7cea32-8481-4054-89c1-6b72c41ea4f7" containerName="payload-pre-processing"
Nov 02 08:36:06.637457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637384    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0e98831c-cc04-405f-a7fb-d8c19bf60bcb" containerName="revoke-keys"
Nov 02 08:36:06.637457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637391    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e98831c-cc04-405f-a7fb-d8c19bf60bcb" containerName="revoke-keys"
Nov 02 08:36:06.637457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637410    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy"
Nov 02 08:36:06.637457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637416    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy"
Nov 02 08:36:06.637457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637427    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy"
Nov 02 08:36:06.637457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637431    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy"
Nov 02 08:36:06.637457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637440    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy"
Nov 02 08:36:06.637457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637445    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy"
Nov 02 08:36:06.637716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637534    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7cea32-8481-4054-89c1-6b72c41ea4f7" containerName="payload-pre-processing"
Nov 02 08:36:06.637716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637545    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e98831c-cc04-405f-a7fb-d8c19bf60bcb" containerName="revoke-keys"
Nov 02 08:36:06.637716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637555    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy"
Nov 02 08:36:06.637716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637561    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy"
Nov 02 08:36:06.637716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637567    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3341400d-3694-48ec-86e8-e0ed90f5776b" containerName="istio-proxy"
Nov 02 08:36:06.637716 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.637574    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy"
Nov 02 08:36:06.642424 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.642408    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:06.657415 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.657389    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:36:06.657540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.657481    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:36:06.657585 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.657550    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:36:06.657637 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.657626    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:36:06.659285 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.659264    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"]
Nov 02 08:36:06.793103 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.793066    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd1807b8-4545h\" (UID: \"938815d3-2163-4f72-9a95-58c1b084bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:06.793287 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.793237    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttvwh\" (UniqueName: \"kubernetes.io/projected/938815d3-2163-4f72-9a95-58c1b084bbf2-kube-api-access-ttvwh\") pod \"maas-api-revoke-sub-bd1807b8-4545h\" (UID: \"938815d3-2163-4f72-9a95-58c1b084bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:06.793464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.793435    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd1807b8-4545h\" (UID: \"938815d3-2163-4f72-9a95-58c1b084bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:06.894347 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.894260    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd1807b8-4545h\" (UID: \"938815d3-2163-4f72-9a95-58c1b084bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:06.894509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.894419    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttvwh\" (UniqueName: \"kubernetes.io/projected/938815d3-2163-4f72-9a95-58c1b084bbf2-kube-api-access-ttvwh\") pod \"maas-api-revoke-sub-bd1807b8-4545h\" (UID: \"938815d3-2163-4f72-9a95-58c1b084bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:06.894592 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.894576    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd1807b8-4545h\" (UID: \"938815d3-2163-4f72-9a95-58c1b084bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:06.895244 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.895215    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd1807b8-4545h\" (UID: \"938815d3-2163-4f72-9a95-58c1b084bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:06.896658 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.896634    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd1807b8-4545h\" (UID: \"938815d3-2163-4f72-9a95-58c1b084bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:06.919946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.919919    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttvwh\" (UniqueName: \"kubernetes.io/projected/938815d3-2163-4f72-9a95-58c1b084bbf2-kube-api-access-ttvwh\") pod \"maas-api-revoke-sub-bd1807b8-4545h\" (UID: \"938815d3-2163-4f72-9a95-58c1b084bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:06.957786 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:06.957754    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:07.284571 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.283723    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"]
Nov 02 08:36:07.288330 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:36:07.288100    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938815d3_2163_4f72_9a95_58c1b084bbf2.slice/crio-460efe038b1e4a0437c0631baa48613d96874cfc0074d59863b3e23716ce496f WatchSource:0}: Error finding container 460efe038b1e4a0437c0631baa48613d96874cfc0074d59863b3e23716ce496f: Status 404 returned error can't find the container with id 460efe038b1e4a0437c0631baa48613d96874cfc0074d59863b3e23716ce496f
Nov 02 08:36:07.346373 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.346347    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 08:36:07.409390 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.409351    4898 generic.go:356] "Generic (PLEG): container finished" podID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerID="21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd" exitCode=0
Nov 02 08:36:07.409390 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.409388    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" event={"ID":"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09","Type":"ContainerDied","Data":"21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd"}
Nov 02 08:36:07.409709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.409410    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"
Nov 02 08:36:07.409709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.409444    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx" event={"ID":"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09","Type":"ContainerDied","Data":"06378a476f1a7507282b3acf5ffa963a05761c7110d458c5634b8764fe5bfe75"}
Nov 02 08:36:07.409709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.409472    4898 scope.go:122] "RemoveContainer" containerID="21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd"
Nov 02 08:36:07.411468 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.411438    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h" event={"ID":"938815d3-2163-4f72-9a95-58c1b084bbf2","Type":"ContainerStarted","Data":"460efe038b1e4a0437c0631baa48613d96874cfc0074d59863b3e23716ce496f"}
Nov 02 08:36:07.421844 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.421808    4898 scope.go:122] "RemoveContainer" containerID="8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456"
Nov 02 08:36:07.432758 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.432734    4898 scope.go:122] "RemoveContainer" containerID="21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd"
Nov 02 08:36:07.433116 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:36:07.433092    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd\": container with ID starting with 21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd not found: ID does not exist" containerID="21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd"
Nov 02 08:36:07.433206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.433126    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd"} err="failed to get container status \"21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd\": rpc error: code = NotFound desc = could not find container \"21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd\": container with ID starting with 21bf9d1a4d290992811fc36fc8b2d776a20d4f0e6215603ed3e62a748b32ebbd not found: ID does not exist"
Nov 02 08:36:07.433206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.433146    4898 scope.go:122] "RemoveContainer" containerID="8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456"
Nov 02 08:36:07.433461 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:36:07.433444    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456\": container with ID starting with 8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456 not found: ID does not exist" containerID="8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456"
Nov 02 08:36:07.433527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.433465    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456"} err="failed to get container status \"8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456\": rpc error: code = NotFound desc = could not find container \"8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456\": container with ID starting with 8ba8e95cfda779d021d03eeb8fe1ff52375d45043aa2782020b4c7679c16a456 not found: ID does not exist"
Nov 02 08:36:07.504455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.504426    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-token\" (UniqueName: \"kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-token\") pod \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") "
Nov 02 08:36:07.504603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.504467    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-certs\") pod \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") "
Nov 02 08:36:07.504603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.504526    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-data\") pod \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") "
Nov 02 08:36:07.504603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.504555    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-socket\") pod \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") "
Nov 02 08:36:07.504603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.504590    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-envoy\") pod \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") "
Nov 02 08:36:07.504835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.504610    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-podinfo\") pod \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") "
Nov 02 08:36:07.504835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.504675    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-credential-socket\") pod \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") "
Nov 02 08:36:07.504835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.504741    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istiod-ca-cert\") pod \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") "
Nov 02 08:36:07.505051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.504834    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-kube-api-access-bv9t8\" (UniqueName: \"kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-kube-api-access-bv9t8\") pod \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\" (UID: \"73b37b0d-d0e2-4f0f-9ac5-fb6318737a09\") "
Nov 02 08:36:07.505051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.504849    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-certs" pod "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" (UID: "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:07.505051 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.504992    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-socket" pod "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" (UID: "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:07.505221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.505101    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-credential-socket" pod "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" (UID: "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:07.505284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.505235    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-data" pod "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" (UID: "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:07.505284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.505241    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istiod-ca-cert" pod "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" (UID: "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:07.505493 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.505460    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:07.505622 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.505495    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:07.505622 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.505511    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:07.505622 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.505524    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:07.505622 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.505539    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:07.506836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.506811    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-token" pod "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" (UID: "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:07.507117 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.507088    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-podinfo" pod "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" (UID: "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:36:07.507301 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.507277    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-envoy" pod "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" (UID: "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:07.507382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.507366    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-kube-api-access-bv9t8" pod "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" (UID: "73b37b0d-d0e2-4f0f-9ac5-fb6318737a09"). InnerVolumeSpecName "kube-api-access-bv9t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:07.607046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.607012    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv9t8\" (UniqueName: \"kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-kube-api-access-bv9t8\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:07.607185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.607048    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:07.607185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.607062    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:07.607185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.607075    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:07.757782 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.757713    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"]
Nov 02 08:36:07.762553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:07.762525    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c712b1-openshift-default-565f6f489-92kfx"]
Nov 02 08:36:08.425549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:08.425511    4898 generic.go:356] "Generic (PLEG): container finished" podID="938815d3-2163-4f72-9a95-58c1b084bbf2" containerID="40af73718f78ee132e00c35ab9768526637ffaa6930fe09df7d325a7f2584df4" exitCode=0
Nov 02 08:36:08.425763 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:08.425639    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h" event={"ID":"938815d3-2163-4f72-9a95-58c1b084bbf2","Type":"ContainerDied","Data":"40af73718f78ee132e00c35ab9768526637ffaa6930fe09df7d325a7f2584df4"}
Nov 02 08:36:08.543857 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:08.543777    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"]
Nov 02 08:36:09.468342 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:09.468302    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" path="/var/lib/kubelet/pods/73b37b0d-d0e2-4f0f-9ac5-fb6318737a09/volumes"
Nov 02 08:36:09.623100 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:09.623069    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:09.738383 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:09.738297    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-xks-ca\") pod \"938815d3-2163-4f72-9a95-58c1b084bbf2\" (UID: \"938815d3-2163-4f72-9a95-58c1b084bbf2\") "
Nov 02 08:36:09.738383 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:09.738362    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-openshift-ca\") pod \"938815d3-2163-4f72-9a95-58c1b084bbf2\" (UID: \"938815d3-2163-4f72-9a95-58c1b084bbf2\") "
Nov 02 08:36:09.738636 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:09.738408    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938815d3-2163-4f72-9a95-58c1b084bbf2-kube-api-access-ttvwh\" (UniqueName: \"kubernetes.io/projected/938815d3-2163-4f72-9a95-58c1b084bbf2-kube-api-access-ttvwh\") pod \"938815d3-2163-4f72-9a95-58c1b084bbf2\" (UID: \"938815d3-2163-4f72-9a95-58c1b084bbf2\") "
Nov 02 08:36:09.738779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:09.738752    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-openshift-ca" pod "938815d3-2163-4f72-9a95-58c1b084bbf2" (UID: "938815d3-2163-4f72-9a95-58c1b084bbf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:09.740477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:09.740447    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-xks-ca" pod "938815d3-2163-4f72-9a95-58c1b084bbf2" (UID: "938815d3-2163-4f72-9a95-58c1b084bbf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:36:09.740683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:09.740660    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938815d3-2163-4f72-9a95-58c1b084bbf2-kube-api-access-ttvwh" pod "938815d3-2163-4f72-9a95-58c1b084bbf2" (UID: "938815d3-2163-4f72-9a95-58c1b084bbf2"). InnerVolumeSpecName "kube-api-access-ttvwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:09.840375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:09.840336    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:09.840375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:09.840368    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938815d3-2163-4f72-9a95-58c1b084bbf2-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:09.840375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:09.840383    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttvwh\" (UniqueName: \"kubernetes.io/projected/938815d3-2163-4f72-9a95-58c1b084bbf2-kube-api-access-ttvwh\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:10.455096 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:10.455062    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"
Nov 02 08:36:10.455293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:10.455088    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h" event={"ID":"938815d3-2163-4f72-9a95-58c1b084bbf2","Type":"ContainerDied","Data":"460efe038b1e4a0437c0631baa48613d96874cfc0074d59863b3e23716ce496f"}
Nov 02 08:36:10.455293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:10.455144    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="460efe038b1e4a0437c0631baa48613d96874cfc0074d59863b3e23716ce496f"
Nov 02 08:36:13.544322 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:13.544284    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb" podUID="8f53dcda-e118-4c9e-ba95-097a44061d18" containerName="payload-processing" containerID="cri-o://79ab56cf2fcc620dde84884af9626041ea66e51e89b65190c50a8ddb7b8e0092" gracePeriod=25
Nov 02 08:36:14.989556 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:36:14.989513    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:36:15.469950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:15.469916    4898 scope.go:122] "RemoveContainer" containerID="3025ba2e8908bd42618de99c4ddd1a93b0d960f40ff1f0cafae1d036b8481bdb"
Nov 02 08:36:16.989860 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.989821    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"]
Nov 02 08:36:16.990501 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.990478    4898 cpu_manager.go:414] "containerMap: removing container" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy"
Nov 02 08:36:16.990616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.990503    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy"
Nov 02 08:36:16.990616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.990521    4898 cpu_manager.go:414] "containerMap: removing container" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy"
Nov 02 08:36:16.990616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.990527    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy"
Nov 02 08:36:16.990616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.990545    4898 cpu_manager.go:414] "containerMap: removing container" podUID="938815d3-2163-4f72-9a95-58c1b084bbf2" containerName="revoke-keys"
Nov 02 08:36:16.990616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.990550    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938815d3-2163-4f72-9a95-58c1b084bbf2" containerName="revoke-keys"
Nov 02 08:36:16.990616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.990584    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy"
Nov 02 08:36:16.990616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.990590    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9609fcde-fbb5-409b-82bf-78eaec870453" containerName="istio-proxy"
Nov 02 08:36:16.991020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.990692    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy"
Nov 02 08:36:16.991020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.990701    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b37b0d-d0e2-4f0f-9ac5-fb6318737a09" containerName="istio-proxy"
Nov 02 08:36:16.991020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.990711    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="938815d3-2163-4f72-9a95-58c1b084bbf2" containerName="revoke-keys"
Nov 02 08:36:16.996206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:16.996173    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:17.013158 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.013130    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"]
Nov 02 08:36:17.020035 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.020000    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:36:17.020212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.020079    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:36:17.020212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.020006    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:36:17.023155 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.023116    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:36:17.134564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.134526    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gz9q\" (UniqueName: \"kubernetes.io/projected/567fbe26-d5e3-498e-a102-1f7ff9e9f031-kube-api-access-8gz9q\") pod \"maas-api-revoke-sub-641991db-k6f96\" (UID: \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:17.134792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.134583    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-xks-ca\") pod \"maas-api-revoke-sub-641991db-k6f96\" (UID: \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:17.134866 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.134784    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-641991db-k6f96\" (UID: \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:17.236315 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.236267    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-641991db-k6f96\" (UID: \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:17.236510 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.236426    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gz9q\" (UniqueName: \"kubernetes.io/projected/567fbe26-d5e3-498e-a102-1f7ff9e9f031-kube-api-access-8gz9q\") pod \"maas-api-revoke-sub-641991db-k6f96\" (UID: \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:17.236510 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.236462    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-xks-ca\") pod \"maas-api-revoke-sub-641991db-k6f96\" (UID: \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:17.237043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.237017    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-641991db-k6f96\" (UID: \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:17.238983 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.238960    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-xks-ca\") pod \"maas-api-revoke-sub-641991db-k6f96\" (UID: \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:17.275841 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.275772    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gz9q\" (UniqueName: \"kubernetes.io/projected/567fbe26-d5e3-498e-a102-1f7ff9e9f031-kube-api-access-8gz9q\") pod \"maas-api-revoke-sub-641991db-k6f96\" (UID: \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:17.315527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.315490    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:17.627993 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:17.627956    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"]
Nov 02 08:36:18.528202 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:18.528158    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96" event={"ID":"567fbe26-d5e3-498e-a102-1f7ff9e9f031","Type":"ContainerStarted","Data":"9d06918414fd2730c1cf0619abfba872f7976dfea7fdc72dc5e85ed753845339"}
Nov 02 08:36:18.528603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:18.528216    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96" event={"ID":"567fbe26-d5e3-498e-a102-1f7ff9e9f031","Type":"ContainerStarted","Data":"8dcefcb8d5c911418aef822872d85d240e29ebfa8943f43f7eb52c4689c73665"}
Nov 02 08:36:18.570400 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:18.570338    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96" podStartSLOduration=2.570323498 podStartE2EDuration="2.570323498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:36:18.568499514 +0000 UTC m=+4329.258826478" watchObservedRunningTime="2026-11-02 08:36:18.570323498 +0000 UTC m=+4329.260650462"
Nov 02 08:36:19.537478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:19.537439    4898 generic.go:356] "Generic (PLEG): container finished" podID="567fbe26-d5e3-498e-a102-1f7ff9e9f031" containerID="9d06918414fd2730c1cf0619abfba872f7976dfea7fdc72dc5e85ed753845339" exitCode=0
Nov 02 08:36:19.537938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:19.537524    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96" event={"ID":"567fbe26-d5e3-498e-a102-1f7ff9e9f031","Type":"ContainerDied","Data":"9d06918414fd2730c1cf0619abfba872f7976dfea7fdc72dc5e85ed753845339"}
Nov 02 08:36:20.696516 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:20.696483    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb" podUID="8f53dcda-e118-4c9e-ba95-097a44061d18" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:20.713202 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:20.713172    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:20.837444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:20.837365    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567fbe26-d5e3-498e-a102-1f7ff9e9f031-kube-api-access-8gz9q\" (UniqueName: \"kubernetes.io/projected/567fbe26-d5e3-498e-a102-1f7ff9e9f031-kube-api-access-8gz9q\") pod \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\" (UID: \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\") "
Nov 02 08:36:20.837604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:20.837493    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-openshift-ca\") pod \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\" (UID: \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\") "
Nov 02 08:36:20.837604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:20.837539    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-xks-ca\") pod \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\" (UID: \"567fbe26-d5e3-498e-a102-1f7ff9e9f031\") "
Nov 02 08:36:20.837973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:20.837946    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-openshift-ca" pod "567fbe26-d5e3-498e-a102-1f7ff9e9f031" (UID: "567fbe26-d5e3-498e-a102-1f7ff9e9f031"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:20.839668 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:20.839646    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567fbe26-d5e3-498e-a102-1f7ff9e9f031-kube-api-access-8gz9q" pod "567fbe26-d5e3-498e-a102-1f7ff9e9f031" (UID: "567fbe26-d5e3-498e-a102-1f7ff9e9f031"). InnerVolumeSpecName "kube-api-access-8gz9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:20.839726 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:20.839711    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-xks-ca" pod "567fbe26-d5e3-498e-a102-1f7ff9e9f031" (UID: "567fbe26-d5e3-498e-a102-1f7ff9e9f031"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:36:20.939208 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:20.939172    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gz9q\" (UniqueName: \"kubernetes.io/projected/567fbe26-d5e3-498e-a102-1f7ff9e9f031-kube-api-access-8gz9q\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:20.939208 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:20.939202    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:20.939208 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:20.939213    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567fbe26-d5e3-498e-a102-1f7ff9e9f031-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:21.554455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:21.554425    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"
Nov 02 08:36:21.554455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:21.554442    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96" event={"ID":"567fbe26-d5e3-498e-a102-1f7ff9e9f031","Type":"ContainerDied","Data":"8dcefcb8d5c911418aef822872d85d240e29ebfa8943f43f7eb52c4689c73665"}
Nov 02 08:36:21.554691 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:21.554489    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dcefcb8d5c911418aef822872d85d240e29ebfa8943f43f7eb52c4689c73665"
Nov 02 08:36:30.696953 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:30.696902    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb" podUID="8f53dcda-e118-4c9e-ba95-097a44061d18" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:32.333444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:32.333404    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"]
Nov 02 08:36:32.333915 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:32.333675    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy" containerID="cri-o://76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f" gracePeriod=30
Nov 02 08:36:32.906063 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:32.906023    4898 patch_prober.go:29] interesting pod/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 08:36:32.906248 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:32.906093    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 08:36:33.671580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:33.671543    4898 generic.go:356] "Generic (PLEG): container finished" podID="8f53dcda-e118-4c9e-ba95-097a44061d18" containerID="79ab56cf2fcc620dde84884af9626041ea66e51e89b65190c50a8ddb7b8e0092" exitCode=0
Nov 02 08:36:33.671994 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:33.671622    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb" event={"ID":"8f53dcda-e118-4c9e-ba95-097a44061d18","Type":"ContainerDied","Data":"79ab56cf2fcc620dde84884af9626041ea66e51e89b65190c50a8ddb7b8e0092"}
Nov 02 08:36:33.848182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:33.848148    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"
Nov 02 08:36:34.032164 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:34.032131    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f53dcda-e118-4c9e-ba95-097a44061d18-kube-api-access-nxsgp\" (UniqueName: \"kubernetes.io/projected/8f53dcda-e118-4c9e-ba95-097a44061d18-kube-api-access-nxsgp\") pod \"8f53dcda-e118-4c9e-ba95-097a44061d18\" (UID: \"8f53dcda-e118-4c9e-ba95-097a44061d18\") "
Nov 02 08:36:34.032361 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:34.032204    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f53dcda-e118-4c9e-ba95-097a44061d18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f53dcda-e118-4c9e-ba95-097a44061d18-plugins-config-volume\") pod \"8f53dcda-e118-4c9e-ba95-097a44061d18\" (UID: \"8f53dcda-e118-4c9e-ba95-097a44061d18\") "
Nov 02 08:36:34.032630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:34.032594    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f53dcda-e118-4c9e-ba95-097a44061d18-plugins-config-volume" pod "8f53dcda-e118-4c9e-ba95-097a44061d18" (UID: "8f53dcda-e118-4c9e-ba95-097a44061d18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:34.034631 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:34.034604    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f53dcda-e118-4c9e-ba95-097a44061d18-kube-api-access-nxsgp" pod "8f53dcda-e118-4c9e-ba95-097a44061d18" (UID: "8f53dcda-e118-4c9e-ba95-097a44061d18"). InnerVolumeSpecName "kube-api-access-nxsgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:34.133353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:34.133267    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxsgp\" (UniqueName: \"kubernetes.io/projected/8f53dcda-e118-4c9e-ba95-097a44061d18-kube-api-access-nxsgp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:34.133353 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:34.133305    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f53dcda-e118-4c9e-ba95-097a44061d18-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:34.681302 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:34.681266    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"
Nov 02 08:36:34.681766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:34.681290    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb" event={"ID":"8f53dcda-e118-4c9e-ba95-097a44061d18","Type":"ContainerDied","Data":"899bf1fcbdc38e95e0a89bf2f68b575e9cfdeb39b8bd65ae4de9dde93fbd7d5a"}
Nov 02 08:36:34.681766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:34.681407    4898 scope.go:122] "RemoveContainer" containerID="79ab56cf2fcc620dde84884af9626041ea66e51e89b65190c50a8ddb7b8e0092"
Nov 02 08:36:34.734431 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:34.734392    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"]
Nov 02 08:36:34.746639 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:34.746601    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bb4907-fb7c6c5cc-vttnb"]
Nov 02 08:36:35.465765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:35.465725    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f53dcda-e118-4c9e-ba95-097a44061d18" path="/var/lib/kubelet/pods/8f53dcda-e118-4c9e-ba95-097a44061d18/volumes"
Nov 02 08:36:36.446784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:36.446750    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"]
Nov 02 08:36:36.491947 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:36.490168    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"]
Nov 02 08:36:37.618969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.618907    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:36:37.676991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.676958    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-socket\") pod \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") "
Nov 02 08:36:37.677187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677021    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-envoy\") pod \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") "
Nov 02 08:36:37.677187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677049    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-credential-socket\") pod \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") "
Nov 02 08:36:37.677187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677101    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-certs\") pod \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") "
Nov 02 08:36:37.677187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677157    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istiod-ca-cert\") pod \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") "
Nov 02 08:36:37.677877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677187    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-podinfo\") pod \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") "
Nov 02 08:36:37.677877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677231    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-kube-api-access-nvfl7\" (UniqueName: \"kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-kube-api-access-nvfl7\") pod \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") "
Nov 02 08:36:37.677877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677262    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-token\" (UniqueName: \"kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-token\") pod \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") "
Nov 02 08:36:37.677877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677360    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-data\") pod \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\" (UID: \"09f1801e-b54e-4616-bb4c-e279bf1c8ee5\") "
Nov 02 08:36:37.677877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677386    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-socket" pod "09f1801e-b54e-4616-bb4c-e279bf1c8ee5" (UID: "09f1801e-b54e-4616-bb4c-e279bf1c8ee5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:37.677877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677399    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-credential-socket" pod "09f1801e-b54e-4616-bb4c-e279bf1c8ee5" (UID: "09f1801e-b54e-4616-bb4c-e279bf1c8ee5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:37.678349 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677899    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:37.678349 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677924    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:37.678349 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.677981    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-certs" pod "09f1801e-b54e-4616-bb4c-e279bf1c8ee5" (UID: "09f1801e-b54e-4616-bb4c-e279bf1c8ee5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:37.678349 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.678169    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-data" pod "09f1801e-b54e-4616-bb4c-e279bf1c8ee5" (UID: "09f1801e-b54e-4616-bb4c-e279bf1c8ee5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:37.678349 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.678088    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istiod-ca-cert" pod "09f1801e-b54e-4616-bb4c-e279bf1c8ee5" (UID: "09f1801e-b54e-4616-bb4c-e279bf1c8ee5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:37.680159 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.680029    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-envoy" pod "09f1801e-b54e-4616-bb4c-e279bf1c8ee5" (UID: "09f1801e-b54e-4616-bb4c-e279bf1c8ee5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:37.681023 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.680514    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-podinfo" pod "09f1801e-b54e-4616-bb4c-e279bf1c8ee5" (UID: "09f1801e-b54e-4616-bb4c-e279bf1c8ee5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:36:37.681023 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.680568    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-token" pod "09f1801e-b54e-4616-bb4c-e279bf1c8ee5" (UID: "09f1801e-b54e-4616-bb4c-e279bf1c8ee5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:37.681023 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.680656    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-kube-api-access-nvfl7" pod "09f1801e-b54e-4616-bb4c-e279bf1c8ee5" (UID: "09f1801e-b54e-4616-bb4c-e279bf1c8ee5"). InnerVolumeSpecName "kube-api-access-nvfl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:37.727625 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.727589    4898 generic.go:356] "Generic (PLEG): container finished" podID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerID="76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f" exitCode=0
Nov 02 08:36:37.727812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.727632    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" event={"ID":"09f1801e-b54e-4616-bb4c-e279bf1c8ee5","Type":"ContainerDied","Data":"76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f"}
Nov 02 08:36:37.727812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.727641    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"
Nov 02 08:36:37.727812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.727676    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc" event={"ID":"09f1801e-b54e-4616-bb4c-e279bf1c8ee5","Type":"ContainerDied","Data":"21336fdf20a0cc39036aac2f727ae906431e24226a14eb59d31275c15a8ad900"}
Nov 02 08:36:37.727812 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.727699    4898 scope.go:122] "RemoveContainer" containerID="76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f"
Nov 02 08:36:37.739490 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.739469    4898 scope.go:122] "RemoveContainer" containerID="dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003"
Nov 02 08:36:37.751486 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.751465    4898 scope.go:122] "RemoveContainer" containerID="76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f"
Nov 02 08:36:37.751786 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:36:37.751760    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f\": container with ID starting with 76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f not found: ID does not exist" containerID="76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f"
Nov 02 08:36:37.751857 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.751798    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f"} err="failed to get container status \"76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f\": rpc error: code = NotFound desc = could not find container \"76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f\": container with ID starting with 76f0c514c309b700338a0684d631170ae56240f11168267da21df592709f095f not found: ID does not exist"
Nov 02 08:36:37.751857 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.751823    4898 scope.go:122] "RemoveContainer" containerID="dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003"
Nov 02 08:36:37.752085 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:36:37.752066    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003\": container with ID starting with dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003 not found: ID does not exist" containerID="dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003"
Nov 02 08:36:37.752135 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.752094    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003"} err="failed to get container status \"dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003\": rpc error: code = NotFound desc = could not find container \"dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003\": container with ID starting with dfcc05bce997f5cb495a66a87de47abf9a43fc740515c6f0a8e734817663b003 not found: ID does not exist"
Nov 02 08:36:37.778967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.778932    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:37.778967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.778965    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:37.779193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.778979    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:37.779193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.778992    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:37.779193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.779008    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvfl7\" (UniqueName: \"kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-kube-api-access-nvfl7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:37.779193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.779020    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:37.779193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.779028    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f1801e-b54e-4616-bb4c-e279bf1c8ee5-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:37.786043 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.786019    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"]
Nov 02 08:36:37.791062 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:37.791036    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bb4907-openshift-default-5d449c9f5c-lj5vc"]
Nov 02 08:36:38.032321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:38.032282    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"]
Nov 02 08:36:38.036201 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:38.036165    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"]
Nov 02 08:36:38.048779 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:38.048721    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb134db2-6gqsh"]
Nov 02 08:36:38.050028 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:38.049999    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150a71d3-bv2xs"]
Nov 02 08:36:38.209022 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:38.208976    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"]
Nov 02 08:36:39.464580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:39.464545    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" path="/var/lib/kubelet/pods/09f1801e-b54e-4616-bb4c-e279bf1c8ee5/volumes"
Nov 02 08:36:39.465079 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:39.465062    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b42405-696b-4aa8-b0e5-22c9556f9ddc" path="/var/lib/kubelet/pods/19b42405-696b-4aa8-b0e5-22c9556f9ddc/volumes"
Nov 02 08:36:39.465488 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:39.465474    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47" path="/var/lib/kubelet/pods/c67d9e0d-ff4d-4ecc-bbd9-e65b064c0d47/volumes"
Nov 02 08:36:40.489910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:40.489858    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"]
Nov 02 08:36:41.447509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:41.447451    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h" podUID="2b3467c1-86b7-479b-8bc3-89e7b0473d89" containerName="payload-processing" containerID="cri-o://6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7" gracePeriod=25
Nov 02 08:36:41.492487 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:41.492444    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw" podUID="13afe874-6c21-463a-b1fb-829e7d104bb8" containerName="payload-pre-processing" containerID="cri-o://0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb" gracePeriod=25
Nov 02 08:36:41.764706 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:41.764612    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw" podUID="13afe874-6c21-463a-b1fb-829e7d104bb8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:41.764924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:41.764623    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h" podUID="2b3467c1-86b7-479b-8bc3-89e7b0473d89" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:44.991220 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:36:44.991181    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:36:45.490276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:45.490232    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl" podUID="0f07eefc-2028-4748-823e-8b0ec22ac61e" containerName="payload-processing" containerID="cri-o://72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee" gracePeriod=25
Nov 02 08:36:48.032366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:48.032321    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"]
Nov 02 08:36:48.035696 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:48.035663    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76fb09e5-4phpf"]
Nov 02 08:36:48.388039 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:48.387945    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl" podUID="0f07eefc-2028-4748-823e-8b0ec22ac61e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:49.498002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:49.497942    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4538bddc-7e27-4ac5-a370-29cf46d6669a" path="/var/lib/kubelet/pods/4538bddc-7e27-4ac5-a370-29cf46d6669a/volumes"
Nov 02 08:36:51.765009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:51.764918    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw" podUID="13afe874-6c21-463a-b1fb-829e7d104bb8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:51.765419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:51.764929    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h" podUID="2b3467c1-86b7-479b-8bc3-89e7b0473d89" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:52.314681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.314639    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"]
Nov 02 08:36:52.315313 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.315289    4898 cpu_manager.go:414] "containerMap: removing container" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy"
Nov 02 08:36:52.315444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.315313    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy"
Nov 02 08:36:52.315444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.315341    4898 cpu_manager.go:414] "containerMap: removing container" podUID="8f53dcda-e118-4c9e-ba95-097a44061d18" containerName="payload-processing"
Nov 02 08:36:52.315444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.315351    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f53dcda-e118-4c9e-ba95-097a44061d18" containerName="payload-processing"
Nov 02 08:36:52.315444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.315365    4898 cpu_manager.go:414] "containerMap: removing container" podUID="567fbe26-d5e3-498e-a102-1f7ff9e9f031" containerName="revoke-keys"
Nov 02 08:36:52.315444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.315372    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567fbe26-d5e3-498e-a102-1f7ff9e9f031" containerName="revoke-keys"
Nov 02 08:36:52.315444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.315415    4898 cpu_manager.go:414] "containerMap: removing container" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy"
Nov 02 08:36:52.315444 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.315423    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy"
Nov 02 08:36:52.315826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.315558    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy"
Nov 02 08:36:52.315826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.315575    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f53dcda-e118-4c9e-ba95-097a44061d18" containerName="payload-processing"
Nov 02 08:36:52.315826 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.315585    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="567fbe26-d5e3-498e-a102-1f7ff9e9f031" containerName="revoke-keys"
Nov 02 08:36:52.319327 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.319302    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:52.338504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.338476    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"]
Nov 02 08:36:52.345349 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.345313    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:36:52.345553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.345324    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:36:52.345729 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.345366    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:36:52.345811 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.345380    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:36:52.497960 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.497920    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e86193d-5nmpz\" (UID: \"47b8fed1-75bc-4c61-a734-0d48d231db82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:52.498162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.498086    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67jtk\" (UniqueName: \"kubernetes.io/projected/47b8fed1-75bc-4c61-a734-0d48d231db82-kube-api-access-67jtk\") pod \"maas-api-revoke-sub-5e86193d-5nmpz\" (UID: \"47b8fed1-75bc-4c61-a734-0d48d231db82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:52.498162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.498156    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e86193d-5nmpz\" (UID: \"47b8fed1-75bc-4c61-a734-0d48d231db82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:52.599156 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.599050    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e86193d-5nmpz\" (UID: \"47b8fed1-75bc-4c61-a734-0d48d231db82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:52.599338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.599194    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67jtk\" (UniqueName: \"kubernetes.io/projected/47b8fed1-75bc-4c61-a734-0d48d231db82-kube-api-access-67jtk\") pod \"maas-api-revoke-sub-5e86193d-5nmpz\" (UID: \"47b8fed1-75bc-4c61-a734-0d48d231db82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:52.599338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.599246    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e86193d-5nmpz\" (UID: \"47b8fed1-75bc-4c61-a734-0d48d231db82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:52.600073 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.600046    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e86193d-5nmpz\" (UID: \"47b8fed1-75bc-4c61-a734-0d48d231db82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:52.601674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.601643    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e86193d-5nmpz\" (UID: \"47b8fed1-75bc-4c61-a734-0d48d231db82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:52.622707 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.622676    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67jtk\" (UniqueName: \"kubernetes.io/projected/47b8fed1-75bc-4c61-a734-0d48d231db82-kube-api-access-67jtk\") pod \"maas-api-revoke-sub-5e86193d-5nmpz\" (UID: \"47b8fed1-75bc-4c61-a734-0d48d231db82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:52.633035 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.633011    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:52.939882 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:52.939849    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"]
Nov 02 08:36:52.940254 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:36:52.940060    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b8fed1_75bc_4c61_a734_0d48d231db82.slice/crio-1ac04e81dfc92215e38d36575b06bb9cf3102e9afa32a5d90554d579c9868c33 WatchSource:0}: Error finding container 1ac04e81dfc92215e38d36575b06bb9cf3102e9afa32a5d90554d579c9868c33: Status 404 returned error can't find the container with id 1ac04e81dfc92215e38d36575b06bb9cf3102e9afa32a5d90554d579c9868c33
Nov 02 08:36:53.030259 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.030224    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"]
Nov 02 08:36:53.034375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.034343    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a36a4e-jsqgq"]
Nov 02 08:36:53.231531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.231434    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k" podUID="ba9ac06a-7a08-4f47-85b6-89061ac53e66" containerName="main" containerID="cri-o://faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7" gracePeriod=45
Nov 02 08:36:53.269613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.269557    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k" podUID="ba9ac06a-7a08-4f47-85b6-89061ac53e66" containerName="main" probeResult="failure" output="Get \"https://10.132.1.152:8000/ready\": dial tcp 10.132.1.152:8000: connect: connection refused"
Nov 02 08:36:53.285250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.285214    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"]
Nov 02 08:36:53.285519 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.285470    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy" containerID="cri-o://b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b" gracePeriod=30
Nov 02 08:36:53.491610 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.491522    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acdd4d3a-3be4-4524-8622-217cf531d6cf" path="/var/lib/kubelet/pods/acdd4d3a-3be4-4524-8622-217cf531d6cf/volumes"
Nov 02 08:36:53.575491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.575456    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:36:53.711707 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.711666    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-dshm\") pod \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") "
Nov 02 08:36:53.711707 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.711713    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-model-cache\") pod \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") "
Nov 02 08:36:53.712036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.711754    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9ac06a-7a08-4f47-85b6-89061ac53e66-kube-api-access-d6vcq\" (UniqueName: \"kubernetes.io/projected/ba9ac06a-7a08-4f47-85b6-89061ac53e66-kube-api-access-d6vcq\") pod \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") "
Nov 02 08:36:53.712036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.711817    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tmp-dir\") pod \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") "
Nov 02 08:36:53.712036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.711854    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tls-certs\") pod \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") "
Nov 02 08:36:53.712235 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.712064    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-home\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-home\") pod \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\" (UID: \"ba9ac06a-7a08-4f47-85b6-89061ac53e66\") "
Nov 02 08:36:53.712235 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.712079    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-model-cache" pod "ba9ac06a-7a08-4f47-85b6-89061ac53e66" (UID: "ba9ac06a-7a08-4f47-85b6-89061ac53e66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:53.712235 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.712083    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tmp-dir" pod "ba9ac06a-7a08-4f47-85b6-89061ac53e66" (UID: "ba9ac06a-7a08-4f47-85b6-89061ac53e66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:53.712356 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.712322    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-home" pod "ba9ac06a-7a08-4f47-85b6-89061ac53e66" (UID: "ba9ac06a-7a08-4f47-85b6-89061ac53e66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:53.712554 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.712533    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:53.712598 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.712558    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:53.712598 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.712571    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:53.714217 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.714190    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-dshm" pod "ba9ac06a-7a08-4f47-85b6-89061ac53e66" (UID: "ba9ac06a-7a08-4f47-85b6-89061ac53e66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:53.714342 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.714234    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tls-certs" pod "ba9ac06a-7a08-4f47-85b6-89061ac53e66" (UID: "ba9ac06a-7a08-4f47-85b6-89061ac53e66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:36:53.714401 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.714350    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9ac06a-7a08-4f47-85b6-89061ac53e66-kube-api-access-d6vcq" pod "ba9ac06a-7a08-4f47-85b6-89061ac53e66" (UID: "ba9ac06a-7a08-4f47-85b6-89061ac53e66"). InnerVolumeSpecName "kube-api-access-d6vcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:53.814077 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.813988    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba9ac06a-7a08-4f47-85b6-89061ac53e66-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:53.814077 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.814023    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6vcq\" (UniqueName: \"kubernetes.io/projected/ba9ac06a-7a08-4f47-85b6-89061ac53e66-kube-api-access-d6vcq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:53.814077 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.814034    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba9ac06a-7a08-4f47-85b6-89061ac53e66-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:53.890175 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.890136    4898 generic.go:356] "Generic (PLEG): container finished" podID="ba9ac06a-7a08-4f47-85b6-89061ac53e66" containerID="faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7" exitCode=0
Nov 02 08:36:53.890175 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.890175    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"
Nov 02 08:36:53.890431 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.890216    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k" event={"ID":"ba9ac06a-7a08-4f47-85b6-89061ac53e66","Type":"ContainerDied","Data":"faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7"}
Nov 02 08:36:53.890431 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.890265    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k" event={"ID":"ba9ac06a-7a08-4f47-85b6-89061ac53e66","Type":"ContainerDied","Data":"3e55bb321411648b82fe185a40ef615ddfbd4a1f4aebd2d3f5844b089482e3da"}
Nov 02 08:36:53.890431 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.890287    4898 scope.go:122] "RemoveContainer" containerID="faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7"
Nov 02 08:36:53.892592 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.892558    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz" event={"ID":"47b8fed1-75bc-4c61-a734-0d48d231db82","Type":"ContainerStarted","Data":"80d636c23a604c8ac922799b0cf030f2b25da8e8ac3f65b7f056370795b0114a"}
Nov 02 08:36:53.892722 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.892619    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz" event={"ID":"47b8fed1-75bc-4c61-a734-0d48d231db82","Type":"ContainerStarted","Data":"1ac04e81dfc92215e38d36575b06bb9cf3102e9afa32a5d90554d579c9868c33"}
Nov 02 08:36:53.902426 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.902406    4898 scope.go:122] "RemoveContainer" containerID="faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7"
Nov 02 08:36:53.902727 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:36:53.902709    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7\": container with ID starting with faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7 not found: ID does not exist" containerID="faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7"
Nov 02 08:36:53.902799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.902735    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7"} err="failed to get container status \"faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7\": rpc error: code = NotFound desc = could not find container \"faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7\": container with ID starting with faff27931913fd81562c6525be847fd9ea7c12add0d0cb2d4014d0153fefb5f7 not found: ID does not exist"
Nov 02 08:36:53.945415 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.945368    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz" podStartSLOduration=1.945356649 podStartE2EDuration="1.945356649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:36:53.944481355 +0000 UTC m=+4364.634808320" watchObservedRunningTime="2026-11-02 08:36:53.945356649 +0000 UTC m=+4364.635683609"
Nov 02 08:36:53.982186 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.982153    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"]
Nov 02 08:36:53.988334 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:53.988303    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a44352/rate-test-model-a44352-kserve-5df4d54b5-t2j9k"]
Nov 02 08:36:54.904813 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:54.904778    4898 generic.go:356] "Generic (PLEG): container finished" podID="47b8fed1-75bc-4c61-a734-0d48d231db82" containerID="80d636c23a604c8ac922799b0cf030f2b25da8e8ac3f65b7f056370795b0114a" exitCode=0
Nov 02 08:36:54.905054 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:54.904860    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz" event={"ID":"47b8fed1-75bc-4c61-a734-0d48d231db82","Type":"ContainerDied","Data":"80d636c23a604c8ac922799b0cf030f2b25da8e8ac3f65b7f056370795b0114a"}
Nov 02 08:36:55.467880 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:55.467837    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9ac06a-7a08-4f47-85b6-89061ac53e66" path="/var/lib/kubelet/pods/ba9ac06a-7a08-4f47-85b6-89061ac53e66/volumes"
Nov 02 08:36:56.071405 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.071374    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:56.144754 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.144695    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b8fed1-75bc-4c61-a734-0d48d231db82-kube-api-access-67jtk\" (UniqueName: \"kubernetes.io/projected/47b8fed1-75bc-4c61-a734-0d48d231db82-kube-api-access-67jtk\") pod \"47b8fed1-75bc-4c61-a734-0d48d231db82\" (UID: \"47b8fed1-75bc-4c61-a734-0d48d231db82\") "
Nov 02 08:36:56.145004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.144936    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-openshift-ca\") pod \"47b8fed1-75bc-4c61-a734-0d48d231db82\" (UID: \"47b8fed1-75bc-4c61-a734-0d48d231db82\") "
Nov 02 08:36:56.145102 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.145021    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-xks-ca\") pod \"47b8fed1-75bc-4c61-a734-0d48d231db82\" (UID: \"47b8fed1-75bc-4c61-a734-0d48d231db82\") "
Nov 02 08:36:56.145571 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.145531    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-openshift-ca" pod "47b8fed1-75bc-4c61-a734-0d48d231db82" (UID: "47b8fed1-75bc-4c61-a734-0d48d231db82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:56.147654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.147627    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b8fed1-75bc-4c61-a734-0d48d231db82-kube-api-access-67jtk" pod "47b8fed1-75bc-4c61-a734-0d48d231db82" (UID: "47b8fed1-75bc-4c61-a734-0d48d231db82"). InnerVolumeSpecName "kube-api-access-67jtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:56.147776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.147643    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-xks-ca" pod "47b8fed1-75bc-4c61-a734-0d48d231db82" (UID: "47b8fed1-75bc-4c61-a734-0d48d231db82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:36:56.246324 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.246231    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67jtk\" (UniqueName: \"kubernetes.io/projected/47b8fed1-75bc-4c61-a734-0d48d231db82-kube-api-access-67jtk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:56.246324 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.246265    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:56.246324 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.246276    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47b8fed1-75bc-4c61-a734-0d48d231db82-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:56.924523 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.924490    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"
Nov 02 08:36:56.925042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.924527    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz" event={"ID":"47b8fed1-75bc-4c61-a734-0d48d231db82","Type":"ContainerDied","Data":"1ac04e81dfc92215e38d36575b06bb9cf3102e9afa32a5d90554d579c9868c33"}
Nov 02 08:36:56.925042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.924581    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ac04e81dfc92215e38d36575b06bb9cf3102e9afa32a5d90554d579c9868c33"
Nov 02 08:36:56.975503 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:56.975469    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"]
Nov 02 08:36:57.002305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:57.002271    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"]
Nov 02 08:36:58.388556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.388505    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl" podUID="0f07eefc-2028-4748-823e-8b0ec22ac61e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:36:58.574460 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.574429    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:36:58.677535 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.677497    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-token\" (UniqueName: \"kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-token\") pod \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") "
Nov 02 08:36:58.677718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.677548    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-envoy\") pod \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") "
Nov 02 08:36:58.677718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.677581    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-podinfo\") pod \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") "
Nov 02 08:36:58.677718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.677666    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-kube-api-access-qfhgw\" (UniqueName: \"kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-kube-api-access-qfhgw\") pod \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") "
Nov 02 08:36:58.677718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.677693    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-data\") pod \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") "
Nov 02 08:36:58.678126 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.677822    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-credential-socket\") pod \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") "
Nov 02 08:36:58.678126 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.677857    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-certs\") pod \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") "
Nov 02 08:36:58.678126 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.677929    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istiod-ca-cert\") pod \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") "
Nov 02 08:36:58.678126 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.677987    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-socket\") pod \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\" (UID: \"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f\") "
Nov 02 08:36:58.678527 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.678497    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-data" pod "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" (UID: "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:58.678745 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.678680    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-socket" pod "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" (UID: "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:58.678884 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.678861    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-credential-socket" pod "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" (UID: "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:58.679345 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.679320    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-certs" pod "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" (UID: "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:58.679770 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.679749    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istiod-ca-cert" pod "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" (UID: "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:36:58.680587 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.680559    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-token" pod "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" (UID: "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:58.681273 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.681229    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-envoy" pod "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" (UID: "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:36:58.681477 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.681447    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-podinfo" pod "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" (UID: "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:36:58.682548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.682523    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-kube-api-access-qfhgw" pod "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" (UID: "dfbb79de-a724-48f3-a6a7-6e40bf71fb6f"). InnerVolumeSpecName "kube-api-access-qfhgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:36:58.779674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.779625    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:58.779674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.779672    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfhgw\" (UniqueName: \"kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-kube-api-access-qfhgw\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:58.779910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.779687    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:58.779910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.779702    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:58.779910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.779714    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:58.779910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.779726    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:58.779910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.779738    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:58.779910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.779752    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:58.779910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.779764    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:36:58.941646 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.941608    4898 generic.go:356] "Generic (PLEG): container finished" podID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerID="b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b" exitCode=0
Nov 02 08:36:58.941835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.941666    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" event={"ID":"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f","Type":"ContainerDied","Data":"b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b"}
Nov 02 08:36:58.941835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.941677    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"
Nov 02 08:36:58.941835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.941734    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4" event={"ID":"dfbb79de-a724-48f3-a6a7-6e40bf71fb6f","Type":"ContainerDied","Data":"4da0c2bfc699ef1ac8df596ef6670a84f50e992ed5b67693ea74bb7f322b0f9e"}
Nov 02 08:36:58.941835 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.941756    4898 scope.go:122] "RemoveContainer" containerID="b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b"
Nov 02 08:36:58.952744 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.952723    4898 scope.go:122] "RemoveContainer" containerID="c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973"
Nov 02 08:36:58.963533 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.963505    4898 scope.go:122] "RemoveContainer" containerID="b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b"
Nov 02 08:36:58.963884 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:36:58.963857    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b\": container with ID starting with b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b not found: ID does not exist" containerID="b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b"
Nov 02 08:36:58.963995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.963910    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b"} err="failed to get container status \"b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b\": rpc error: code = NotFound desc = could not find container \"b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b\": container with ID starting with b6af360d8c252918ac2253c8f4888d60882d074be1e0ec336351fee49a154f8b not found: ID does not exist"
Nov 02 08:36:58.963995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.963939    4898 scope.go:122] "RemoveContainer" containerID="c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973"
Nov 02 08:36:58.964248 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:36:58.964226    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973\": container with ID starting with c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973 not found: ID does not exist" containerID="c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973"
Nov 02 08:36:58.964310 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.964257    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973"} err="failed to get container status \"c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973\": rpc error: code = NotFound desc = could not find container \"c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973\": container with ID starting with c6b702551d3673c4bce440be055472096029ec813b5b389f47f3b394707ac973 not found: ID does not exist"
Nov 02 08:36:58.997909 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:58.997862    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"]
Nov 02 08:36:59.001246 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:59.001223    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3d8293-openshift-default-6d79f978df-7g5q4"]
Nov 02 08:36:59.030998 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:59.030961    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"]
Nov 02 08:36:59.034202 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:59.034178    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dc33abe-lhrkm"]
Nov 02 08:36:59.466305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:59.466269    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d2d129-9f29-4ba8-8995-5e6442461130" path="/var/lib/kubelet/pods/04d2d129-9f29-4ba8-8995-5e6442461130/volumes"
Nov 02 08:36:59.466772 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:36:59.466757    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" path="/var/lib/kubelet/pods/dfbb79de-a724-48f3-a6a7-6e40bf71fb6f/volumes"
Nov 02 08:37:01.801567 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.801535    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"
Nov 02 08:37:01.847231 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.847202    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"
Nov 02 08:37:01.927575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.927541    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b3467c1-86b7-479b-8bc3-89e7b0473d89-kube-api-access-wst85\" (UniqueName: \"kubernetes.io/projected/2b3467c1-86b7-479b-8bc3-89e7b0473d89-kube-api-access-wst85\") pod \"2b3467c1-86b7-479b-8bc3-89e7b0473d89\" (UID: \"2b3467c1-86b7-479b-8bc3-89e7b0473d89\") "
Nov 02 08:37:01.927805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.927611    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13afe874-6c21-463a-b1fb-829e7d104bb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13afe874-6c21-463a-b1fb-829e7d104bb8-plugins-config-volume\") pod \"13afe874-6c21-463a-b1fb-829e7d104bb8\" (UID: \"13afe874-6c21-463a-b1fb-829e7d104bb8\") "
Nov 02 08:37:01.927805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.927786    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13afe874-6c21-463a-b1fb-829e7d104bb8-kube-api-access-q2q2b\" (UniqueName: \"kubernetes.io/projected/13afe874-6c21-463a-b1fb-829e7d104bb8-kube-api-access-q2q2b\") pod \"13afe874-6c21-463a-b1fb-829e7d104bb8\" (UID: \"13afe874-6c21-463a-b1fb-829e7d104bb8\") "
Nov 02 08:37:01.927974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.927840    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b3467c1-86b7-479b-8bc3-89e7b0473d89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3467c1-86b7-479b-8bc3-89e7b0473d89-plugins-config-volume\") pod \"2b3467c1-86b7-479b-8bc3-89e7b0473d89\" (UID: \"2b3467c1-86b7-479b-8bc3-89e7b0473d89\") "
Nov 02 08:37:01.928182 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.928140    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13afe874-6c21-463a-b1fb-829e7d104bb8-plugins-config-volume" pod "13afe874-6c21-463a-b1fb-829e7d104bb8" (UID: "13afe874-6c21-463a-b1fb-829e7d104bb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:01.928446 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.928422    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3467c1-86b7-479b-8bc3-89e7b0473d89-plugins-config-volume" pod "2b3467c1-86b7-479b-8bc3-89e7b0473d89" (UID: "2b3467c1-86b7-479b-8bc3-89e7b0473d89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:01.928551 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.928535    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13afe874-6c21-463a-b1fb-829e7d104bb8-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:01.928623 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.928558    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3467c1-86b7-479b-8bc3-89e7b0473d89-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:01.929990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.929962    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3467c1-86b7-479b-8bc3-89e7b0473d89-kube-api-access-wst85" pod "2b3467c1-86b7-479b-8bc3-89e7b0473d89" (UID: "2b3467c1-86b7-479b-8bc3-89e7b0473d89"). InnerVolumeSpecName "kube-api-access-wst85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:01.930504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.930484    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13afe874-6c21-463a-b1fb-829e7d104bb8-kube-api-access-q2q2b" pod "13afe874-6c21-463a-b1fb-829e7d104bb8" (UID: "13afe874-6c21-463a-b1fb-829e7d104bb8"). InnerVolumeSpecName "kube-api-access-q2q2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:01.964484 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.964455    4898 generic.go:356] "Generic (PLEG): container finished" podID="2b3467c1-86b7-479b-8bc3-89e7b0473d89" containerID="6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7" exitCode=0
Nov 02 08:37:01.964651 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.964502    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"
Nov 02 08:37:01.964651 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.964543    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h" event={"ID":"2b3467c1-86b7-479b-8bc3-89e7b0473d89","Type":"ContainerDied","Data":"6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7"}
Nov 02 08:37:01.964651 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.964606    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h" event={"ID":"2b3467c1-86b7-479b-8bc3-89e7b0473d89","Type":"ContainerDied","Data":"25982b38fdb72201774386519e5e51f00834b47a41c37d9ae2532cf2f61f4d18"}
Nov 02 08:37:01.964651 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.964636    4898 scope.go:122] "RemoveContainer" containerID="6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7"
Nov 02 08:37:01.966132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.966113    4898 generic.go:356] "Generic (PLEG): container finished" podID="13afe874-6c21-463a-b1fb-829e7d104bb8" containerID="0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb" exitCode=0
Nov 02 08:37:01.966221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.966159    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"
Nov 02 08:37:01.966221 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.966186    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw" event={"ID":"13afe874-6c21-463a-b1fb-829e7d104bb8","Type":"ContainerDied","Data":"0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb"}
Nov 02 08:37:01.966305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.966236    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw" event={"ID":"13afe874-6c21-463a-b1fb-829e7d104bb8","Type":"ContainerDied","Data":"dc9e9e89b4bb9fd373546affc95181a19afeeffcdaec6ed62f6d60c237a1d3c4"}
Nov 02 08:37:01.977869 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.977846    4898 scope.go:122] "RemoveContainer" containerID="6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7"
Nov 02 08:37:01.978226 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:01.978204    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7\": container with ID starting with 6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7 not found: ID does not exist" containerID="6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7"
Nov 02 08:37:01.978276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.978234    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7"} err="failed to get container status \"6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7\": rpc error: code = NotFound desc = could not find container \"6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7\": container with ID starting with 6e824ca8bf2ef467fa987d78b1422b5e74f16c0188c2a18ff7416b3fc6b4e6d7 not found: ID does not exist"
Nov 02 08:37:01.978276 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.978253    4898 scope.go:122] "RemoveContainer" containerID="0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb"
Nov 02 08:37:01.987765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.987743    4898 scope.go:122] "RemoveContainer" containerID="0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb"
Nov 02 08:37:01.988193 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:01.988156    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb\": container with ID starting with 0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb not found: ID does not exist" containerID="0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb"
Nov 02 08:37:01.988292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:01.988204    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb"} err="failed to get container status \"0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb\": rpc error: code = NotFound desc = could not find container \"0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb\": container with ID starting with 0d88bfd51acaf4911d4d2cd37e189f6f0d01a787502ff54a51c67499fff986cb not found: ID does not exist"
Nov 02 08:37:02.022404 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:02.022308    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"]
Nov 02 08:37:02.026048 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:02.026021    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw"]
Nov 02 08:37:02.029375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:02.029352    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2q2b\" (UniqueName: \"kubernetes.io/projected/13afe874-6c21-463a-b1fb-829e7d104bb8-kube-api-access-q2q2b\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:02.029375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:02.029376    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wst85\" (UniqueName: \"kubernetes.io/projected/2b3467c1-86b7-479b-8bc3-89e7b0473d89-kube-api-access-wst85\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:02.060956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:02.060910    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"]
Nov 02 08:37:02.067288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:02.067253    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h"]
Nov 02 08:37:02.764181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:02.764061    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56bbb6-548f5c74cf-xdt7h" podUID="2b3467c1-86b7-479b-8bc3-89e7b0473d89" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.213:50052\" within 1s: context deadline exceeded"
Nov 02 08:37:02.764181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:02.764060    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56bbb6-757d5dfd7-t7mgw" podUID="13afe874-6c21-463a-b1fb-829e7d104bb8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.212:50052\" within 1s: context deadline exceeded"
Nov 02 08:37:02.764390 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:37:02.764154    4898 logging.go:55] [core] [Channel #18085 SubChannel #18087]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.212:50052", ServerName: "10.132.0.212:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.212:50052: operation was canceled"
Nov 02 08:37:02.764390 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:37:02.764155    4898 logging.go:55] [core] [Channel #18084 SubChannel #18086]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.213:50052", ServerName: "10.132.0.213:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.213:50052: operation was canceled"
Nov 02 08:37:03.494171 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:03.494066    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13afe874-6c21-463a-b1fb-829e7d104bb8" path="/var/lib/kubelet/pods/13afe874-6c21-463a-b1fb-829e7d104bb8/volumes"
Nov 02 08:37:03.494853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:03.494706    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b3467c1-86b7-479b-8bc3-89e7b0473d89" path="/var/lib/kubelet/pods/2b3467c1-86b7-479b-8bc3-89e7b0473d89/volumes"
Nov 02 08:37:04.115148 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.115108    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"]
Nov 02 08:37:04.115436 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.115403    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy" containerID="cri-o://596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba" gracePeriod=30
Nov 02 08:37:04.794409 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.794371    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"]
Nov 02 08:37:04.794874 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.794861    4898 cpu_manager.go:414] "containerMap: removing container" podUID="13afe874-6c21-463a-b1fb-829e7d104bb8" containerName="payload-pre-processing"
Nov 02 08:37:04.794957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.794875    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13afe874-6c21-463a-b1fb-829e7d104bb8" containerName="payload-pre-processing"
Nov 02 08:37:04.794957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.794902    4898 cpu_manager.go:414] "containerMap: removing container" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy"
Nov 02 08:37:04.794957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.794908    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy"
Nov 02 08:37:04.794957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.794922    4898 cpu_manager.go:414] "containerMap: removing container" podUID="2b3467c1-86b7-479b-8bc3-89e7b0473d89" containerName="payload-processing"
Nov 02 08:37:04.794957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.794927    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b3467c1-86b7-479b-8bc3-89e7b0473d89" containerName="payload-processing"
Nov 02 08:37:04.794957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.794948    4898 cpu_manager.go:414] "containerMap: removing container" podUID="47b8fed1-75bc-4c61-a734-0d48d231db82" containerName="revoke-keys"
Nov 02 08:37:04.794957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.794953    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b8fed1-75bc-4c61-a734-0d48d231db82" containerName="revoke-keys"
Nov 02 08:37:04.795191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.794979    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ba9ac06a-7a08-4f47-85b6-89061ac53e66" containerName="main"
Nov 02 08:37:04.795191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.794983    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9ac06a-7a08-4f47-85b6-89061ac53e66" containerName="main"
Nov 02 08:37:04.795191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.795063    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9ac06a-7a08-4f47-85b6-89061ac53e66" containerName="main"
Nov 02 08:37:04.795191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.795074    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="13afe874-6c21-463a-b1fb-829e7d104bb8" containerName="payload-pre-processing"
Nov 02 08:37:04.795191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.795080    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy"
Nov 02 08:37:04.795191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.795087    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b3467c1-86b7-479b-8bc3-89e7b0473d89" containerName="payload-processing"
Nov 02 08:37:04.795191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.795095    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b8fed1-75bc-4c61-a734-0d48d231db82" containerName="revoke-keys"
Nov 02 08:37:04.795191 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.795102    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f1801e-b54e-4616-bb4c-e279bf1c8ee5" containerName="istio-proxy"
Nov 02 08:37:04.799920 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.799897    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:04.817347 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.817303    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"]
Nov 02 08:37:04.825288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.825261    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:37:04.825437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.825377    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:37:04.825437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.825383    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:37:04.827055 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.827029    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:37:04.863122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.863082    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk2lx\" (UniqueName: \"kubernetes.io/projected/34165fb1-dda1-403c-9fc7-0251e8769a5e-kube-api-access-dk2lx\") pod \"maas-api-revoke-sub-dd7d3351-b2m8h\" (UID: \"34165fb1-dda1-403c-9fc7-0251e8769a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:04.863287 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.863209    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd7d3351-b2m8h\" (UID: \"34165fb1-dda1-403c-9fc7-0251e8769a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:04.863287 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.863245    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd7d3351-b2m8h\" (UID: \"34165fb1-dda1-403c-9fc7-0251e8769a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:04.964469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.964431    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk2lx\" (UniqueName: \"kubernetes.io/projected/34165fb1-dda1-403c-9fc7-0251e8769a5e-kube-api-access-dk2lx\") pod \"maas-api-revoke-sub-dd7d3351-b2m8h\" (UID: \"34165fb1-dda1-403c-9fc7-0251e8769a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:04.964654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.964515    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd7d3351-b2m8h\" (UID: \"34165fb1-dda1-403c-9fc7-0251e8769a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:04.964654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.964548    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd7d3351-b2m8h\" (UID: \"34165fb1-dda1-403c-9fc7-0251e8769a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:04.965268 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.965248    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd7d3351-b2m8h\" (UID: \"34165fb1-dda1-403c-9fc7-0251e8769a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:04.967018 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.966990    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd7d3351-b2m8h\" (UID: \"34165fb1-dda1-403c-9fc7-0251e8769a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:04.988504 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:04.988476    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk2lx\" (UniqueName: \"kubernetes.io/projected/34165fb1-dda1-403c-9fc7-0251e8769a5e-kube-api-access-dk2lx\") pod \"maas-api-revoke-sub-dd7d3351-b2m8h\" (UID: \"34165fb1-dda1-403c-9fc7-0251e8769a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:05.114678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:05.114584    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:05.444205 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:05.440117    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"]
Nov 02 08:37:05.752786 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:05.752758    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"
Nov 02 08:37:05.876508 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:05.876473    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f07eefc-2028-4748-823e-8b0ec22ac61e-kube-api-access-dbr7p\" (UniqueName: \"kubernetes.io/projected/0f07eefc-2028-4748-823e-8b0ec22ac61e-kube-api-access-dbr7p\") pod \"0f07eefc-2028-4748-823e-8b0ec22ac61e\" (UID: \"0f07eefc-2028-4748-823e-8b0ec22ac61e\") "
Nov 02 08:37:05.877037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:05.876580    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f07eefc-2028-4748-823e-8b0ec22ac61e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f07eefc-2028-4748-823e-8b0ec22ac61e-plugins-config-volume\") pod \"0f07eefc-2028-4748-823e-8b0ec22ac61e\" (UID: \"0f07eefc-2028-4748-823e-8b0ec22ac61e\") "
Nov 02 08:37:05.877037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:05.877023    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f07eefc-2028-4748-823e-8b0ec22ac61e-plugins-config-volume" pod "0f07eefc-2028-4748-823e-8b0ec22ac61e" (UID: "0f07eefc-2028-4748-823e-8b0ec22ac61e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:05.877168 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:05.877121    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f07eefc-2028-4748-823e-8b0ec22ac61e-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:05.878744 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:05.878720    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f07eefc-2028-4748-823e-8b0ec22ac61e-kube-api-access-dbr7p" pod "0f07eefc-2028-4748-823e-8b0ec22ac61e" (UID: "0f07eefc-2028-4748-823e-8b0ec22ac61e"). InnerVolumeSpecName "kube-api-access-dbr7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:05.977840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:05.977806    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbr7p\" (UniqueName: \"kubernetes.io/projected/0f07eefc-2028-4748-823e-8b0ec22ac61e-kube-api-access-dbr7p\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:06.001464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:06.001368    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h" event={"ID":"34165fb1-dda1-403c-9fc7-0251e8769a5e","Type":"ContainerStarted","Data":"d50b7c962381a0ef63c107723cbc266b51948720f5960488ef2b6ce5e46d2f88"}
Nov 02 08:37:06.001464 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:06.001437    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h" event={"ID":"34165fb1-dda1-403c-9fc7-0251e8769a5e","Type":"ContainerStarted","Data":"f72bbca8114b3b7a0182d60f658d729b1903b08f53daa4079a6fa10b6a2c1b0a"}
Nov 02 08:37:06.002802 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:06.002777    4898 generic.go:356] "Generic (PLEG): container finished" podID="0f07eefc-2028-4748-823e-8b0ec22ac61e" containerID="72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee" exitCode=0
Nov 02 08:37:06.002940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:06.002816    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"
Nov 02 08:37:06.002940 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:06.002819    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl" event={"ID":"0f07eefc-2028-4748-823e-8b0ec22ac61e","Type":"ContainerDied","Data":"72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee"}
Nov 02 08:37:06.003085 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:06.002966    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl" event={"ID":"0f07eefc-2028-4748-823e-8b0ec22ac61e","Type":"ContainerDied","Data":"822a204aed6cf9b6367910a5eced737aa4146fb30c2a48b4c68d1899ae566861"}
Nov 02 08:37:06.003085 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:06.003007    4898 scope.go:122] "RemoveContainer" containerID="72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee"
Nov 02 08:37:06.013639 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:06.013616    4898 scope.go:122] "RemoveContainer" containerID="72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee"
Nov 02 08:37:06.013944 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:06.013920    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee\": container with ID starting with 72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee not found: ID does not exist" containerID="72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee"
Nov 02 08:37:06.014020 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:06.013958    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee"} err="failed to get container status \"72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee\": rpc error: code = NotFound desc = could not find container \"72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee\": container with ID starting with 72fcfa596704e75c0a1ce115c3565896767a61744319994c9ef0bf58ae9db1ee not found: ID does not exist"
Nov 02 08:37:06.043904 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:06.043841    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h" podStartSLOduration=2.043828234 podStartE2EDuration="2.043828234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:37:06.041108137 +0000 UTC m=+4376.731435120" watchObservedRunningTime="2026-11-02 08:37:06.043828234 +0000 UTC m=+4376.734155197"
Nov 02 08:37:06.078381 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:06.078337    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"]
Nov 02 08:37:06.086279 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:06.086245    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9877ce-6f64cd4bff-tfpwl"]
Nov 02 08:37:07.012593 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:07.012554    4898 generic.go:356] "Generic (PLEG): container finished" podID="34165fb1-dda1-403c-9fc7-0251e8769a5e" containerID="d50b7c962381a0ef63c107723cbc266b51948720f5960488ef2b6ce5e46d2f88" exitCode=0
Nov 02 08:37:07.013018 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:07.012615    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h" event={"ID":"34165fb1-dda1-403c-9fc7-0251e8769a5e","Type":"ContainerDied","Data":"d50b7c962381a0ef63c107723cbc266b51948720f5960488ef2b6ce5e46d2f88"}
Nov 02 08:37:07.029718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:07.029685    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"]
Nov 02 08:37:07.041595 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:07.041565    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeac7505-d4r54"]
Nov 02 08:37:07.467771 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:07.467733    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d85230-4346-4ab7-a102-2944191e0a81" path="/var/lib/kubelet/pods/07d85230-4346-4ab7-a102-2944191e0a81/volumes"
Nov 02 08:37:07.468379 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:07.468360    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f07eefc-2028-4748-823e-8b0ec22ac61e" path="/var/lib/kubelet/pods/0f07eefc-2028-4748-823e-8b0ec22ac61e/volumes"
Nov 02 08:37:08.169356 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:08.169324    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:08.302506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:08.302408    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-openshift-ca\") pod \"34165fb1-dda1-403c-9fc7-0251e8769a5e\" (UID: \"34165fb1-dda1-403c-9fc7-0251e8769a5e\") "
Nov 02 08:37:08.302506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:08.302478    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-xks-ca\") pod \"34165fb1-dda1-403c-9fc7-0251e8769a5e\" (UID: \"34165fb1-dda1-403c-9fc7-0251e8769a5e\") "
Nov 02 08:37:08.302752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:08.302610    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34165fb1-dda1-403c-9fc7-0251e8769a5e-kube-api-access-dk2lx\" (UniqueName: \"kubernetes.io/projected/34165fb1-dda1-403c-9fc7-0251e8769a5e-kube-api-access-dk2lx\") pod \"34165fb1-dda1-403c-9fc7-0251e8769a5e\" (UID: \"34165fb1-dda1-403c-9fc7-0251e8769a5e\") "
Nov 02 08:37:08.302821 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:08.302791    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-openshift-ca" pod "34165fb1-dda1-403c-9fc7-0251e8769a5e" (UID: "34165fb1-dda1-403c-9fc7-0251e8769a5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:08.303063 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:08.303041    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:08.305012 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:08.304979    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-xks-ca" pod "34165fb1-dda1-403c-9fc7-0251e8769a5e" (UID: "34165fb1-dda1-403c-9fc7-0251e8769a5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:08.305136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:08.305031    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34165fb1-dda1-403c-9fc7-0251e8769a5e-kube-api-access-dk2lx" pod "34165fb1-dda1-403c-9fc7-0251e8769a5e" (UID: "34165fb1-dda1-403c-9fc7-0251e8769a5e"). InnerVolumeSpecName "kube-api-access-dk2lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:08.404113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:08.404076    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34165fb1-dda1-403c-9fc7-0251e8769a5e-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:08.404113 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:08.404108    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk2lx\" (UniqueName: \"kubernetes.io/projected/34165fb1-dda1-403c-9fc7-0251e8769a5e-kube-api-access-dk2lx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:09.028948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.028918    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"
Nov 02 08:37:09.028948 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.028928    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h" event={"ID":"34165fb1-dda1-403c-9fc7-0251e8769a5e","Type":"ContainerDied","Data":"f72bbca8114b3b7a0182d60f658d729b1903b08f53daa4079a6fa10b6a2c1b0a"}
Nov 02 08:37:09.029178 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.028979    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f72bbca8114b3b7a0182d60f658d729b1903b08f53daa4079a6fa10b6a2c1b0a"
Nov 02 08:37:09.385515 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.385478    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 08:37:09.414002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.413966    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-credential-socket\") pod \"0d446180-273b-4b72-9dcf-e0979ddbe681\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") "
Nov 02 08:37:09.414219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414011    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-kube-api-access-5lws9\" (UniqueName: \"kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-kube-api-access-5lws9\") pod \"0d446180-273b-4b72-9dcf-e0979ddbe681\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") "
Nov 02 08:37:09.414219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414058    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-podinfo\") pod \"0d446180-273b-4b72-9dcf-e0979ddbe681\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") "
Nov 02 08:37:09.414219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414119    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-envoy\") pod \"0d446180-273b-4b72-9dcf-e0979ddbe681\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") "
Nov 02 08:37:09.414219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414150    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-socket\") pod \"0d446180-273b-4b72-9dcf-e0979ddbe681\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") "
Nov 02 08:37:09.414219 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414176    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-certs\") pod \"0d446180-273b-4b72-9dcf-e0979ddbe681\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") "
Nov 02 08:37:09.414511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414293    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-data\") pod \"0d446180-273b-4b72-9dcf-e0979ddbe681\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") "
Nov 02 08:37:09.414511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414313    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-credential-socket" pod "0d446180-273b-4b72-9dcf-e0979ddbe681" (UID: "0d446180-273b-4b72-9dcf-e0979ddbe681"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:09.414511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414370    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d446180-273b-4b72-9dcf-e0979ddbe681-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d446180-273b-4b72-9dcf-e0979ddbe681-istiod-ca-cert\") pod \"0d446180-273b-4b72-9dcf-e0979ddbe681\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") "
Nov 02 08:37:09.414511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414409    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-token\" (UniqueName: \"kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-token\") pod \"0d446180-273b-4b72-9dcf-e0979ddbe681\" (UID: \"0d446180-273b-4b72-9dcf-e0979ddbe681\") "
Nov 02 08:37:09.414511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414496    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-socket" pod "0d446180-273b-4b72-9dcf-e0979ddbe681" (UID: "0d446180-273b-4b72-9dcf-e0979ddbe681"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:09.414780 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414659    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-data" pod "0d446180-273b-4b72-9dcf-e0979ddbe681" (UID: "0d446180-273b-4b72-9dcf-e0979ddbe681"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:09.414939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414867    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d446180-273b-4b72-9dcf-e0979ddbe681-istiod-ca-cert" pod "0d446180-273b-4b72-9dcf-e0979ddbe681" (UID: "0d446180-273b-4b72-9dcf-e0979ddbe681"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:09.414939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414916    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:09.414939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414939    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:09.415164 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414953    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:09.415164 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.414955    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-certs" pod "0d446180-273b-4b72-9dcf-e0979ddbe681" (UID: "0d446180-273b-4b72-9dcf-e0979ddbe681"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:09.416530 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.416504    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-podinfo" pod "0d446180-273b-4b72-9dcf-e0979ddbe681" (UID: "0d446180-273b-4b72-9dcf-e0979ddbe681"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:37:09.416707 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.416689    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-token" pod "0d446180-273b-4b72-9dcf-e0979ddbe681" (UID: "0d446180-273b-4b72-9dcf-e0979ddbe681"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:09.416824 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.416807    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-kube-api-access-5lws9" pod "0d446180-273b-4b72-9dcf-e0979ddbe681" (UID: "0d446180-273b-4b72-9dcf-e0979ddbe681"). InnerVolumeSpecName "kube-api-access-5lws9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:09.417003 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.416815    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-envoy" pod "0d446180-273b-4b72-9dcf-e0979ddbe681" (UID: "0d446180-273b-4b72-9dcf-e0979ddbe681"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:09.516670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.516356    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d446180-273b-4b72-9dcf-e0979ddbe681-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:09.516670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.516391    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:09.516670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.516407    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lws9\" (UniqueName: \"kubernetes.io/projected/0d446180-273b-4b72-9dcf-e0979ddbe681-kube-api-access-5lws9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:09.516670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.516421    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:09.516670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.516437    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:09.516670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:09.516459    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d446180-273b-4b72-9dcf-e0979ddbe681-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:10.038315 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.038272    4898 generic.go:356] "Generic (PLEG): container finished" podID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerID="596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba" exitCode=0
Nov 02 08:37:10.038540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.038334    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"
Nov 02 08:37:10.038540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.038362    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" event={"ID":"0d446180-273b-4b72-9dcf-e0979ddbe681","Type":"ContainerDied","Data":"596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba"}
Nov 02 08:37:10.038540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.038433    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z" event={"ID":"0d446180-273b-4b72-9dcf-e0979ddbe681","Type":"ContainerDied","Data":"007d77fd20a7fe55561107b2fe094a0abf6f0f91a0d000a5baa6e090e0942e28"}
Nov 02 08:37:10.038540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.038466    4898 scope.go:122] "RemoveContainer" containerID="596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba"
Nov 02 08:37:10.040101 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.040076    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"]
Nov 02 08:37:10.053995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.053973    4898 scope.go:122] "RemoveContainer" containerID="4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a"
Nov 02 08:37:10.054882 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.054861    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc354ad3-5hjjq"]
Nov 02 08:37:10.064729 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.064707    4898 scope.go:122] "RemoveContainer" containerID="596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba"
Nov 02 08:37:10.065135 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:10.065108    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba\": container with ID starting with 596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba not found: ID does not exist" containerID="596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba"
Nov 02 08:37:10.065260 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.065146    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba"} err="failed to get container status \"596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba\": rpc error: code = NotFound desc = could not find container \"596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba\": container with ID starting with 596168a11d8d859225322363476fd8cc68141204ff87435234bc660073f7f0ba not found: ID does not exist"
Nov 02 08:37:10.065260 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.065173    4898 scope.go:122] "RemoveContainer" containerID="4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a"
Nov 02 08:37:10.065611 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:10.065590    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a\": container with ID starting with 4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a not found: ID does not exist" containerID="4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a"
Nov 02 08:37:10.065710 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.065645    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a"} err="failed to get container status \"4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a\": rpc error: code = NotFound desc = could not find container \"4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a\": container with ID starting with 4a307682738f949827acff92d69c9bcf4a7180825e7f75ce160cd02996c15a1a not found: ID does not exist"
Nov 02 08:37:10.120050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.120005    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"]
Nov 02 08:37:10.129614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:10.129578    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9877ce-openshift-default-68479d46df-2f96z"]
Nov 02 08:37:11.466439 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:11.466397    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" path="/var/lib/kubelet/pods/0d446180-273b-4b72-9dcf-e0979ddbe681/volumes"
Nov 02 08:37:11.467171 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:11.467146    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850a7958-bf3e-4176-8e57-29001545b7be" path="/var/lib/kubelet/pods/850a7958-bf3e-4176-8e57-29001545b7be/volumes"
Nov 02 08:37:11.997614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:11.997550    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main" containerID="cri-o://8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c" gracePeriod=45
Nov 02 08:37:12.024314 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.023961    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main" containerID="cri-o://52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735" gracePeriod=45
Nov 02 08:37:12.385564 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.385478    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 08:37:12.387953 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.387922    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 08:37:12.475924 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.475867    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-model-cache\") pod \"cd11c253-5683-46aa-9b84-40ad7d77303b\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") "
Nov 02 08:37:12.477112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.475975    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-kube-api-access-blfhw\" (UniqueName: \"kubernetes.io/projected/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-kube-api-access-blfhw\") pod \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") "
Nov 02 08:37:12.477112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.476026    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-home\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-home\") pod \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") "
Nov 02 08:37:12.477112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.476095    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tmp-dir\") pod \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") "
Nov 02 08:37:12.477112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.476132    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-home\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-home\") pod \"cd11c253-5683-46aa-9b84-40ad7d77303b\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") "
Nov 02 08:37:12.477112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.476158    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-dshm\") pod \"cd11c253-5683-46aa-9b84-40ad7d77303b\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") "
Nov 02 08:37:12.477112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.477060    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd11c253-5683-46aa-9b84-40ad7d77303b-kube-api-access-gr9j2\" (UniqueName: \"kubernetes.io/projected/cd11c253-5683-46aa-9b84-40ad7d77303b-kube-api-access-gr9j2\") pod \"cd11c253-5683-46aa-9b84-40ad7d77303b\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") "
Nov 02 08:37:12.477112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.477111    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-model-cache\") pod \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") "
Nov 02 08:37:12.477805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.477152    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tls-certs\" (UniqueName: \"kubernetes.io/secret/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tls-certs\") pod \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") "
Nov 02 08:37:12.477805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.477240    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd11c253-5683-46aa-9b84-40ad7d77303b-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd11c253-5683-46aa-9b84-40ad7d77303b-tls-certs\") pod \"cd11c253-5683-46aa-9b84-40ad7d77303b\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") "
Nov 02 08:37:12.477805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.477349    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-dshm\") pod \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\" (UID: \"71cd64dc-bdb6-4d4b-ac78-bf67f4795702\") "
Nov 02 08:37:12.477805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.477408    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-tmp-dir\") pod \"cd11c253-5683-46aa-9b84-40ad7d77303b\" (UID: \"cd11c253-5683-46aa-9b84-40ad7d77303b\") "
Nov 02 08:37:12.478193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.478163    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-tmp-dir" pod "cd11c253-5683-46aa-9b84-40ad7d77303b" (UID: "cd11c253-5683-46aa-9b84-40ad7d77303b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:12.478574 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.478546    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-model-cache" pod "cd11c253-5683-46aa-9b84-40ad7d77303b" (UID: "cd11c253-5683-46aa-9b84-40ad7d77303b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:12.479830 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.479509    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-home" pod "cd11c253-5683-46aa-9b84-40ad7d77303b" (UID: "cd11c253-5683-46aa-9b84-40ad7d77303b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:12.479830 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.479598    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-home" pod "71cd64dc-bdb6-4d4b-ac78-bf67f4795702" (UID: "71cd64dc-bdb6-4d4b-ac78-bf67f4795702"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:12.480668 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.480554    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-model-cache" pod "71cd64dc-bdb6-4d4b-ac78-bf67f4795702" (UID: "71cd64dc-bdb6-4d4b-ac78-bf67f4795702"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:12.481137 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.480854    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tmp-dir" pod "71cd64dc-bdb6-4d4b-ac78-bf67f4795702" (UID: "71cd64dc-bdb6-4d4b-ac78-bf67f4795702"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:12.481752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.481719    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tls-certs" pod "71cd64dc-bdb6-4d4b-ac78-bf67f4795702" (UID: "71cd64dc-bdb6-4d4b-ac78-bf67f4795702"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:12.484534 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.484500    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-dshm" pod "cd11c253-5683-46aa-9b84-40ad7d77303b" (UID: "cd11c253-5683-46aa-9b84-40ad7d77303b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:12.485196 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.485158    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd11c253-5683-46aa-9b84-40ad7d77303b-tls-certs" pod "cd11c253-5683-46aa-9b84-40ad7d77303b" (UID: "cd11c253-5683-46aa-9b84-40ad7d77303b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:12.485300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.485252    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-kube-api-access-blfhw" pod "71cd64dc-bdb6-4d4b-ac78-bf67f4795702" (UID: "71cd64dc-bdb6-4d4b-ac78-bf67f4795702"). InnerVolumeSpecName "kube-api-access-blfhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:12.485467 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.485446    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd11c253-5683-46aa-9b84-40ad7d77303b-kube-api-access-gr9j2" pod "cd11c253-5683-46aa-9b84-40ad7d77303b" (UID: "cd11c253-5683-46aa-9b84-40ad7d77303b"). InnerVolumeSpecName "kube-api-access-gr9j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:12.491596 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.491558    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-dshm" pod "71cd64dc-bdb6-4d4b-ac78-bf67f4795702" (UID: "71cd64dc-bdb6-4d4b-ac78-bf67f4795702"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:12.579000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.578838    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.579000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.578871    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.579000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.578882    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.579000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.578924    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.579000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.578939    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9j2\" (UniqueName: \"kubernetes.io/projected/cd11c253-5683-46aa-9b84-40ad7d77303b-kube-api-access-gr9j2\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.579000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.578954    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.579000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.578967    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.579000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.578997    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd11c253-5683-46aa-9b84-40ad7d77303b-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.579447 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.579020    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.579447 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.579031    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.579447 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.579043    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd11c253-5683-46aa-9b84-40ad7d77303b-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:12.579447 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:12.579057    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blfhw\" (UniqueName: \"kubernetes.io/projected/71cd64dc-bdb6-4d4b-ac78-bf67f4795702-kube-api-access-blfhw\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:13.065490 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.065452    4898 generic.go:356] "Generic (PLEG): container finished" podID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerID="52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735" exitCode=0
Nov 02 08:37:13.065700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.065499    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"
Nov 02 08:37:13.065700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.065538    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" event={"ID":"cd11c253-5683-46aa-9b84-40ad7d77303b","Type":"ContainerDied","Data":"52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735"}
Nov 02 08:37:13.065700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.065579    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr" event={"ID":"cd11c253-5683-46aa-9b84-40ad7d77303b","Type":"ContainerDied","Data":"4e3a3363e7812264fa2c68ffed651a831a70c8d961ebfb5ef271aa0a330dc601"}
Nov 02 08:37:13.065700 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.065601    4898 scope.go:122] "RemoveContainer" containerID="52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735"
Nov 02 08:37:13.067414 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.067392    4898 generic.go:356] "Generic (PLEG): container finished" podID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerID="8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c" exitCode=0
Nov 02 08:37:13.067551 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.067445    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"
Nov 02 08:37:13.067551 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.067480    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" event={"ID":"71cd64dc-bdb6-4d4b-ac78-bf67f4795702","Type":"ContainerDied","Data":"8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c"}
Nov 02 08:37:13.067551 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.067521    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc" event={"ID":"71cd64dc-bdb6-4d4b-ac78-bf67f4795702","Type":"ContainerDied","Data":"e4a4bc3d0f96db0675bae2ec5d4effcf954e1a5d507070bd5678339bc948a09b"}
Nov 02 08:37:13.079047 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.079026    4898 scope.go:122] "RemoveContainer" containerID="52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735"
Nov 02 08:37:13.079348 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:13.079325    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735\": container with ID starting with 52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735 not found: ID does not exist" containerID="52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735"
Nov 02 08:37:13.079429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.079360    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735"} err="failed to get container status \"52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735\": rpc error: code = NotFound desc = could not find container \"52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735\": container with ID starting with 52613dbdcbc26da9364c9535f5d072b0f1badce1692ac4a65a0247e1f7c4e735 not found: ID does not exist"
Nov 02 08:37:13.079429 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.079377    4898 scope.go:122] "RemoveContainer" containerID="8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c"
Nov 02 08:37:13.090227 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.090201    4898 scope.go:122] "RemoveContainer" containerID="8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c"
Nov 02 08:37:13.090534 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:13.090510    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c\": container with ID starting with 8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c not found: ID does not exist" containerID="8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c"
Nov 02 08:37:13.090616 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.090540    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c"} err="failed to get container status \"8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c\": rpc error: code = NotFound desc = could not find container \"8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c\": container with ID starting with 8f4a5aad2df5f107321c5e1c2ed0605541703dda190a522b35307ed00212cd1c not found: ID does not exist"
Nov 02 08:37:13.120375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.120334    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"]
Nov 02 08:37:13.133168 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.133136    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-354660/simulator-w3-354660-kserve-5f5cb69cdb-2xlgr"]
Nov 02 08:37:13.160550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.160512    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"]
Nov 02 08:37:13.164060 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.164029    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-354660/distinct-2-w3-354660-kserve-86c6cff557-xnrtc"]
Nov 02 08:37:13.466068 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.466032    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" path="/var/lib/kubelet/pods/71cd64dc-bdb6-4d4b-ac78-bf67f4795702/volumes"
Nov 02 08:37:13.466658 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:13.466489    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" path="/var/lib/kubelet/pods/cd11c253-5683-46aa-9b84-40ad7d77303b/volumes"
Nov 02 08:37:14.585830 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:14.585795    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"]
Nov 02 08:37:14.990264 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:14.990221    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:37:18.035579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:18.035524    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"]
Nov 02 08:37:18.047731 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:18.047699    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb475eb-jwcxh"]
Nov 02 08:37:19.464800 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:19.464764    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29227432-f91a-4c80-9010-8c8c313a145c" path="/var/lib/kubelet/pods/29227432-f91a-4c80-9010-8c8c313a145c/volumes"
Nov 02 08:37:19.586546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:19.586487    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk" podUID="961445c8-a7a3-464c-846c-fb355e661f6e" containerName="payload-pre-processing" containerID="cri-o://9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822" gracePeriod=25
Nov 02 08:37:21.221966 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:21.221917    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk" podUID="961445c8-a7a3-464c-846c-fb355e661f6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:22.300141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.300086    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"]
Nov 02 08:37:22.301633 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.301599    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy"
Nov 02 08:37:22.301836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.301821    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy"
Nov 02 08:37:22.301967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.301954    4898 cpu_manager.go:414] "containerMap: removing container" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main"
Nov 02 08:37:22.302069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302058    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main"
Nov 02 08:37:22.302185 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302175    4898 cpu_manager.go:414] "containerMap: removing container" podUID="34165fb1-dda1-403c-9fc7-0251e8769a5e" containerName="revoke-keys"
Nov 02 08:37:22.302258 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302248    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34165fb1-dda1-403c-9fc7-0251e8769a5e" containerName="revoke-keys"
Nov 02 08:37:22.302352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302327    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0f07eefc-2028-4748-823e-8b0ec22ac61e" containerName="payload-processing"
Nov 02 08:37:22.302352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302342    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f07eefc-2028-4748-823e-8b0ec22ac61e" containerName="payload-processing"
Nov 02 08:37:22.302352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302354    4898 cpu_manager.go:414] "containerMap: removing container" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy"
Nov 02 08:37:22.302606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302362    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy"
Nov 02 08:37:22.302606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302396    4898 cpu_manager.go:414] "containerMap: removing container" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main"
Nov 02 08:37:22.302606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302403    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main"
Nov 02 08:37:22.302606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302529    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="34165fb1-dda1-403c-9fc7-0251e8769a5e" containerName="revoke-keys"
Nov 02 08:37:22.302606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302549    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f07eefc-2028-4748-823e-8b0ec22ac61e" containerName="payload-processing"
Nov 02 08:37:22.302606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302559    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbb79de-a724-48f3-a6a7-6e40bf71fb6f" containerName="istio-proxy"
Nov 02 08:37:22.302606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302575    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd11c253-5683-46aa-9b84-40ad7d77303b" containerName="main"
Nov 02 08:37:22.302606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302590    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy"
Nov 02 08:37:22.302606 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302601    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cd64dc-bdb6-4d4b-ac78-bf67f4795702" containerName="main"
Nov 02 08:37:22.303116 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.302616    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy"
Nov 02 08:37:22.307040 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.307015    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:22.322041 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.322012    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"]
Nov 02 08:37:22.330410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.330379    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:37:22.330610 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.330394    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:37:22.370099 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.370058    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:37:22.370099 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.370089    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:37:22.405639 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.405605    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81019a41-qk6qk\" (UID: \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:22.405832 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.405729    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81019a41-qk6qk\" (UID: \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:22.405926 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.405830    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzbcf\" (UniqueName: \"kubernetes.io/projected/308d8aaf-05e2-4c02-9a17-2b5442e9a188-kube-api-access-zzbcf\") pod \"maas-api-revoke-sub-81019a41-qk6qk\" (UID: \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:22.506883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.506845    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzbcf\" (UniqueName: \"kubernetes.io/projected/308d8aaf-05e2-4c02-9a17-2b5442e9a188-kube-api-access-zzbcf\") pod \"maas-api-revoke-sub-81019a41-qk6qk\" (UID: \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:22.507081 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.506946    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81019a41-qk6qk\" (UID: \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:22.507081 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.507018    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81019a41-qk6qk\" (UID: \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:22.507807 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.507778    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81019a41-qk6qk\" (UID: \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:22.509653 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.509631    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81019a41-qk6qk\" (UID: \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:22.530124 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.530101    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzbcf\" (UniqueName: \"kubernetes.io/projected/308d8aaf-05e2-4c02-9a17-2b5442e9a188-kube-api-access-zzbcf\") pod \"maas-api-revoke-sub-81019a41-qk6qk\" (UID: \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:22.623198 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.623061    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:22.943522 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:22.943489    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"]
Nov 02 08:37:22.944431 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:37:22.944398    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308d8aaf_05e2_4c02_9a17_2b5442e9a188.slice/crio-8be65ef69d7de3d79736f751ccc97ac30d19d5d99fde6007a46e77d08300036a WatchSource:0}: Error finding container 8be65ef69d7de3d79736f751ccc97ac30d19d5d99fde6007a46e77d08300036a: Status 404 returned error can't find the container with id 8be65ef69d7de3d79736f751ccc97ac30d19d5d99fde6007a46e77d08300036a
Nov 02 08:37:23.147962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:23.147918    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk" event={"ID":"308d8aaf-05e2-4c02-9a17-2b5442e9a188","Type":"ContainerStarted","Data":"8be65ef69d7de3d79736f751ccc97ac30d19d5d99fde6007a46e77d08300036a"}
Nov 02 08:37:24.155595 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:24.155559    4898 generic.go:356] "Generic (PLEG): container finished" podID="308d8aaf-05e2-4c02-9a17-2b5442e9a188" containerID="e535f7425024c8b82fd883d7c3f7a85547a7cd0fe5ef8d420b20cee397c2f8b4" exitCode=0
Nov 02 08:37:24.155991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:24.155653    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk" event={"ID":"308d8aaf-05e2-4c02-9a17-2b5442e9a188","Type":"ContainerDied","Data":"e535f7425024c8b82fd883d7c3f7a85547a7cd0fe5ef8d420b20cee397c2f8b4"}
Nov 02 08:37:25.322553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:25.322519    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:25.440131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:25.440043    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-openshift-ca\") pod \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\" (UID: \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\") "
Nov 02 08:37:25.440318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:25.440207    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-xks-ca\") pod \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\" (UID: \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\") "
Nov 02 08:37:25.440318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:25.440233    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308d8aaf-05e2-4c02-9a17-2b5442e9a188-kube-api-access-zzbcf\" (UniqueName: \"kubernetes.io/projected/308d8aaf-05e2-4c02-9a17-2b5442e9a188-kube-api-access-zzbcf\") pod \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\" (UID: \"308d8aaf-05e2-4c02-9a17-2b5442e9a188\") "
Nov 02 08:37:25.440476 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:25.440447    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-openshift-ca" pod "308d8aaf-05e2-4c02-9a17-2b5442e9a188" (UID: "308d8aaf-05e2-4c02-9a17-2b5442e9a188"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:25.440625 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:25.440609    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:25.442354 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:25.442325    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-xks-ca" pod "308d8aaf-05e2-4c02-9a17-2b5442e9a188" (UID: "308d8aaf-05e2-4c02-9a17-2b5442e9a188"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:25.442483 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:25.442428    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308d8aaf-05e2-4c02-9a17-2b5442e9a188-kube-api-access-zzbcf" pod "308d8aaf-05e2-4c02-9a17-2b5442e9a188" (UID: "308d8aaf-05e2-4c02-9a17-2b5442e9a188"). InnerVolumeSpecName "kube-api-access-zzbcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:25.541847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:25.541819    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308d8aaf-05e2-4c02-9a17-2b5442e9a188-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:25.541847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:25.541845    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzbcf\" (UniqueName: \"kubernetes.io/projected/308d8aaf-05e2-4c02-9a17-2b5442e9a188-kube-api-access-zzbcf\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:26.184854 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:26.184808    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk" event={"ID":"308d8aaf-05e2-4c02-9a17-2b5442e9a188","Type":"ContainerDied","Data":"8be65ef69d7de3d79736f751ccc97ac30d19d5d99fde6007a46e77d08300036a"}
Nov 02 08:37:26.185194 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:26.185175    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8be65ef69d7de3d79736f751ccc97ac30d19d5d99fde6007a46e77d08300036a"
Nov 02 08:37:26.185363 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:26.185350    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"
Nov 02 08:37:26.721502 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:26.721462    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"]
Nov 02 08:37:30.870153 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:30.870110    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"]
Nov 02 08:37:30.907845 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:30.907804    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"]
Nov 02 08:37:31.220996 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:31.220958    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk" podUID="961445c8-a7a3-464c-846c-fb355e661f6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:33.611273 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.611237    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"]
Nov 02 08:37:33.611750 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.611735    4898 cpu_manager.go:414] "containerMap: removing container" podUID="308d8aaf-05e2-4c02-9a17-2b5442e9a188" containerName="revoke-keys"
Nov 02 08:37:33.611799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.611752    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="308d8aaf-05e2-4c02-9a17-2b5442e9a188" containerName="revoke-keys"
Nov 02 08:37:33.611799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.611769    4898 cpu_manager.go:414] "containerMap: removing container" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy"
Nov 02 08:37:33.611799 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.611774    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d446180-273b-4b72-9dcf-e0979ddbe681" containerName="istio-proxy"
Nov 02 08:37:33.611945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.611868    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="308d8aaf-05e2-4c02-9a17-2b5442e9a188" containerName="revoke-keys"
Nov 02 08:37:33.615376 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.615355    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:33.632991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.632966    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"]
Nov 02 08:37:33.641651 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.641631    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:37:33.641651 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.641637    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:37:33.641802 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.641653    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:37:33.643425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.643407    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:37:33.741784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.741744    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6e80bf-djb7x\" (UID: \"c11b851c-efbb-4f8b-8dab-d37b56776c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:33.741976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.741817    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qsp\" (UniqueName: \"kubernetes.io/projected/c11b851c-efbb-4f8b-8dab-d37b56776c86-kube-api-access-r5qsp\") pod \"maas-api-revoke-sub-4c6e80bf-djb7x\" (UID: \"c11b851c-efbb-4f8b-8dab-d37b56776c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:33.742029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.741995    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6e80bf-djb7x\" (UID: \"c11b851c-efbb-4f8b-8dab-d37b56776c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:33.842672 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.842638    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6e80bf-djb7x\" (UID: \"c11b851c-efbb-4f8b-8dab-d37b56776c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:33.842881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.842680    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qsp\" (UniqueName: \"kubernetes.io/projected/c11b851c-efbb-4f8b-8dab-d37b56776c86-kube-api-access-r5qsp\") pod \"maas-api-revoke-sub-4c6e80bf-djb7x\" (UID: \"c11b851c-efbb-4f8b-8dab-d37b56776c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:33.842881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.842744    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6e80bf-djb7x\" (UID: \"c11b851c-efbb-4f8b-8dab-d37b56776c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:33.843400 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.843376    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6e80bf-djb7x\" (UID: \"c11b851c-efbb-4f8b-8dab-d37b56776c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:33.845070 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.845045    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6e80bf-djb7x\" (UID: \"c11b851c-efbb-4f8b-8dab-d37b56776c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:33.867224 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.867158    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qsp\" (UniqueName: \"kubernetes.io/projected/c11b851c-efbb-4f8b-8dab-d37b56776c86-kube-api-access-r5qsp\") pod \"maas-api-revoke-sub-4c6e80bf-djb7x\" (UID: \"c11b851c-efbb-4f8b-8dab-d37b56776c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:33.929883 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:33.929851    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:34.231822 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:34.231786    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"]
Nov 02 08:37:34.232416 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:37:34.232379    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11b851c_efbb_4f8b_8dab_d37b56776c86.slice/crio-9717eac1f0f597bee32c0bdffc18eb4923877a2e313a5035e733a83ea3000325 WatchSource:0}: Error finding container 9717eac1f0f597bee32c0bdffc18eb4923877a2e313a5035e733a83ea3000325: Status 404 returned error can't find the container with id 9717eac1f0f597bee32c0bdffc18eb4923877a2e313a5035e733a83ea3000325
Nov 02 08:37:34.246842 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:34.246812    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x" event={"ID":"c11b851c-efbb-4f8b-8dab-d37b56776c86","Type":"ContainerStarted","Data":"9717eac1f0f597bee32c0bdffc18eb4923877a2e313a5035e733a83ea3000325"}
Nov 02 08:37:34.638754 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:34.638720    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"]
Nov 02 08:37:34.675208 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:34.675181    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"]
Nov 02 08:37:35.263945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:35.263062    4898 generic.go:356] "Generic (PLEG): container finished" podID="c11b851c-efbb-4f8b-8dab-d37b56776c86" containerID="0fbda3ada91666107a119c96feb34f4bd6b5ce5a6994f801136357b3d5f9b68c" exitCode=0
Nov 02 08:37:35.263945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:35.263507    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x" event={"ID":"c11b851c-efbb-4f8b-8dab-d37b56776c86","Type":"ContainerDied","Data":"0fbda3ada91666107a119c96feb34f4bd6b5ce5a6994f801136357b3d5f9b68c"}
Nov 02 08:37:35.870947 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:35.870868    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv" podUID="c3566a46-846d-46a7-86e5-26159d70d0dc" containerName="payload-processing" containerID="cri-o://8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e" gracePeriod=25
Nov 02 08:37:35.909953 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:35.908965    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z" podUID="8f1541ea-41ea-4c23-8f9c-9279f17592ca" containerName="payload-pre-processing" containerID="cri-o://2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9" gracePeriod=25
Nov 02 08:37:36.444359 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:36.444331    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:36.475791 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:36.475764    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-xks-ca\") pod \"c11b851c-efbb-4f8b-8dab-d37b56776c86\" (UID: \"c11b851c-efbb-4f8b-8dab-d37b56776c86\") "
Nov 02 08:37:36.475973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:36.475876    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-openshift-ca\") pod \"c11b851c-efbb-4f8b-8dab-d37b56776c86\" (UID: \"c11b851c-efbb-4f8b-8dab-d37b56776c86\") "
Nov 02 08:37:36.475973 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:36.475966    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c11b851c-efbb-4f8b-8dab-d37b56776c86-kube-api-access-r5qsp\" (UniqueName: \"kubernetes.io/projected/c11b851c-efbb-4f8b-8dab-d37b56776c86-kube-api-access-r5qsp\") pod \"c11b851c-efbb-4f8b-8dab-d37b56776c86\" (UID: \"c11b851c-efbb-4f8b-8dab-d37b56776c86\") "
Nov 02 08:37:36.476299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:36.476258    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-openshift-ca" pod "c11b851c-efbb-4f8b-8dab-d37b56776c86" (UID: "c11b851c-efbb-4f8b-8dab-d37b56776c86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:36.476420 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:36.476391    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:36.478622 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:36.478596    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-xks-ca" pod "c11b851c-efbb-4f8b-8dab-d37b56776c86" (UID: "c11b851c-efbb-4f8b-8dab-d37b56776c86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:36.478734 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:36.478619    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11b851c-efbb-4f8b-8dab-d37b56776c86-kube-api-access-r5qsp" pod "c11b851c-efbb-4f8b-8dab-d37b56776c86" (UID: "c11b851c-efbb-4f8b-8dab-d37b56776c86"). InnerVolumeSpecName "kube-api-access-r5qsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:36.578022 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:36.577933    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5qsp\" (UniqueName: \"kubernetes.io/projected/c11b851c-efbb-4f8b-8dab-d37b56776c86-kube-api-access-r5qsp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:36.578022 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:36.577971    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11b851c-efbb-4f8b-8dab-d37b56776c86-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:37.281740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:37.281703    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x" event={"ID":"c11b851c-efbb-4f8b-8dab-d37b56776c86","Type":"ContainerDied","Data":"9717eac1f0f597bee32c0bdffc18eb4923877a2e313a5035e733a83ea3000325"}
Nov 02 08:37:37.281740 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:37.281733    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"
Nov 02 08:37:37.282286 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:37.281760    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9717eac1f0f597bee32c0bdffc18eb4923877a2e313a5035e733a83ea3000325"
Nov 02 08:37:38.433459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:38.433418    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z" podUID="8f1541ea-41ea-4c23-8f9c-9279f17592ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:38.433982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:38.433435    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv" podUID="c3566a46-846d-46a7-86e5-26159d70d0dc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:39.070197 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:39.070161    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"]
Nov 02 08:37:39.070398 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:39.070367    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" podUID="f9683de3-4678-406d-bd92-5cad8ae2637b" containerName="istio-proxy" containerID="cri-o://8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c" gracePeriod=30
Nov 02 08:37:39.676378 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:39.676337    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4" podUID="eea92ad5-8888-44e8-b546-a58d39a3d37e" containerName="payload-pre-processing" containerID="cri-o://e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b" gracePeriod=25
Nov 02 08:37:39.873766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:39.873736    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"
Nov 02 08:37:39.918072 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:39.918033    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961445c8-a7a3-464c-846c-fb355e661f6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961445c8-a7a3-464c-846c-fb355e661f6e-plugins-config-volume\") pod \"961445c8-a7a3-464c-846c-fb355e661f6e\" (UID: \"961445c8-a7a3-464c-846c-fb355e661f6e\") "
Nov 02 08:37:39.918325 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:39.918195    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961445c8-a7a3-464c-846c-fb355e661f6e-kube-api-access-fqk2n\" (UniqueName: \"kubernetes.io/projected/961445c8-a7a3-464c-846c-fb355e661f6e-kube-api-access-fqk2n\") pod \"961445c8-a7a3-464c-846c-fb355e661f6e\" (UID: \"961445c8-a7a3-464c-846c-fb355e661f6e\") "
Nov 02 08:37:39.918515 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:39.918488    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961445c8-a7a3-464c-846c-fb355e661f6e-plugins-config-volume" pod "961445c8-a7a3-464c-846c-fb355e661f6e" (UID: "961445c8-a7a3-464c-846c-fb355e661f6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:39.921204 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:39.921177    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961445c8-a7a3-464c-846c-fb355e661f6e-kube-api-access-fqk2n" pod "961445c8-a7a3-464c-846c-fb355e661f6e" (UID: "961445c8-a7a3-464c-846c-fb355e661f6e"). InnerVolumeSpecName "kube-api-access-fqk2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:40.019603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.019562    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqk2n\" (UniqueName: \"kubernetes.io/projected/961445c8-a7a3-464c-846c-fb355e661f6e-kube-api-access-fqk2n\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:40.019603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.019605    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961445c8-a7a3-464c-846c-fb355e661f6e-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:40.263917 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.263846    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99" podUID="d031a556-bb96-4947-bdc9-a96b70bac427" containerName="payload-processing" containerID="cri-o://d7229b4c958a98d898db1c50c8554295b82abbe01b1271e6f6f9f9c523c9bd1b" gracePeriod=25
Nov 02 08:37:40.304037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.303998    4898 generic.go:356] "Generic (PLEG): container finished" podID="961445c8-a7a3-464c-846c-fb355e661f6e" containerID="9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822" exitCode=0
Nov 02 08:37:40.304199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.304054    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"
Nov 02 08:37:40.304199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.304088    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk" event={"ID":"961445c8-a7a3-464c-846c-fb355e661f6e","Type":"ContainerDied","Data":"9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822"}
Nov 02 08:37:40.304199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.304152    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk" event={"ID":"961445c8-a7a3-464c-846c-fb355e661f6e","Type":"ContainerDied","Data":"c633efe76c760766054d398e3b99c6e55248717ccae2296b8ca993081dcf4b0b"}
Nov 02 08:37:40.304199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.304188    4898 scope.go:122] "RemoveContainer" containerID="9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822"
Nov 02 08:37:40.315406 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.315388    4898 scope.go:122] "RemoveContainer" containerID="9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822"
Nov 02 08:37:40.315704 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:40.315683    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822\": container with ID starting with 9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822 not found: ID does not exist" containerID="9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822"
Nov 02 08:37:40.315770 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.315712    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822"} err="failed to get container status \"9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822\": rpc error: code = NotFound desc = could not find container \"9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822\": container with ID starting with 9a1ec796e732e77fcf8a286524cb089898d89ec73709f71e162634c6ea711822 not found: ID does not exist"
Nov 02 08:37:40.360255 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.360221    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"]
Nov 02 08:37:40.367539 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.367510    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea9556-6d7cb9c4bc-kzcmk"]
Nov 02 08:37:40.438211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.438175    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4" podUID="eea92ad5-8888-44e8-b546-a58d39a3d37e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:40.438388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:40.438347    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99" podUID="d031a556-bb96-4947-bdc9-a96b70bac427" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:41.307286 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.307244    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"]
Nov 02 08:37:41.307814 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.307793    4898 cpu_manager.go:414] "containerMap: removing container" podUID="961445c8-a7a3-464c-846c-fb355e661f6e" containerName="payload-pre-processing"
Nov 02 08:37:41.307873 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.307817    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961445c8-a7a3-464c-846c-fb355e661f6e" containerName="payload-pre-processing"
Nov 02 08:37:41.307951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.307902    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c11b851c-efbb-4f8b-8dab-d37b56776c86" containerName="revoke-keys"
Nov 02 08:37:41.307951 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.307911    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11b851c-efbb-4f8b-8dab-d37b56776c86" containerName="revoke-keys"
Nov 02 08:37:41.308052 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.308040    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11b851c-efbb-4f8b-8dab-d37b56776c86" containerName="revoke-keys"
Nov 02 08:37:41.308093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.308057    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="961445c8-a7a3-464c-846c-fb355e661f6e" containerName="payload-pre-processing"
Nov 02 08:37:41.313373 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.313352    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:41.334004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.333964    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgxq\" (UniqueName: \"kubernetes.io/projected/459a5dd3-a78c-4ba7-9338-e36624d2584c-kube-api-access-9xgxq\") pod \"maas-api-revoke-sub-b0c7f302-fl54g\" (UID: \"459a5dd3-a78c-4ba7-9338-e36624d2584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:41.334156 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.334040    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c7f302-fl54g\" (UID: \"459a5dd3-a78c-4ba7-9338-e36624d2584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:41.334220 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.334175    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c7f302-fl54g\" (UID: \"459a5dd3-a78c-4ba7-9338-e36624d2584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:41.338199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.338167    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"]
Nov 02 08:37:41.338328 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.338213    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:37:41.338328 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.338317    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:37:41.338466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.338361    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:37:41.345476 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.345444    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8895v\""
Nov 02 08:37:41.435622 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.435578    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c7f302-fl54g\" (UID: \"459a5dd3-a78c-4ba7-9338-e36624d2584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:41.435832 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.435657    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgxq\" (UniqueName: \"kubernetes.io/projected/459a5dd3-a78c-4ba7-9338-e36624d2584c-kube-api-access-9xgxq\") pod \"maas-api-revoke-sub-b0c7f302-fl54g\" (UID: \"459a5dd3-a78c-4ba7-9338-e36624d2584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:41.435832 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.435685    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c7f302-fl54g\" (UID: \"459a5dd3-a78c-4ba7-9338-e36624d2584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:41.436557 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.436527    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c7f302-fl54g\" (UID: \"459a5dd3-a78c-4ba7-9338-e36624d2584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:41.438195 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.438169    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c7f302-fl54g\" (UID: \"459a5dd3-a78c-4ba7-9338-e36624d2584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:41.458863 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.458834    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgxq\" (UniqueName: \"kubernetes.io/projected/459a5dd3-a78c-4ba7-9338-e36624d2584c-kube-api-access-9xgxq\") pod \"maas-api-revoke-sub-b0c7f302-fl54g\" (UID: \"459a5dd3-a78c-4ba7-9338-e36624d2584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:41.464471 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.464440    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961445c8-a7a3-464c-846c-fb355e661f6e" path="/var/lib/kubelet/pods/961445c8-a7a3-464c-846c-fb355e661f6e/volumes"
Nov 02 08:37:41.626721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.626646    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:41.744293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.744239    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main" containerID="cri-o://93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98" gracePeriod=45
Nov 02 08:37:41.941287 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:41.941248    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"]
Nov 02 08:37:41.941628 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:37:41.941599    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459a5dd3_a78c_4ba7_9338_e36624d2584c.slice/crio-16203a734e9693cd2e89fbe22804d662bc9fffb1f66e9d21edf6b5b6451ffb82 WatchSource:0}: Error finding container 16203a734e9693cd2e89fbe22804d662bc9fffb1f66e9d21edf6b5b6451ffb82: Status 404 returned error can't find the container with id 16203a734e9693cd2e89fbe22804d662bc9fffb1f66e9d21edf6b5b6451ffb82
Nov 02 08:37:42.009004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.008980    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 08:37:42.042075 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.042049    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-home\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-home\") pod \"1947df5d-7c32-4906-9e5b-266da776ed8b\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") "
Nov 02 08:37:42.042257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.042151    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-model-cache\") pod \"1947df5d-7c32-4906-9e5b-266da776ed8b\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") "
Nov 02 08:37:42.042257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.042183    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1947df5d-7c32-4906-9e5b-266da776ed8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1947df5d-7c32-4906-9e5b-266da776ed8b-tls-certs\") pod \"1947df5d-7c32-4906-9e5b-266da776ed8b\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") "
Nov 02 08:37:42.042257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.042204    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-tmp-dir\") pod \"1947df5d-7c32-4906-9e5b-266da776ed8b\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") "
Nov 02 08:37:42.042257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.042227    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-dshm\") pod \"1947df5d-7c32-4906-9e5b-266da776ed8b\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") "
Nov 02 08:37:42.042506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.042296    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1947df5d-7c32-4906-9e5b-266da776ed8b-kube-api-access-tsmgm\" (UniqueName: \"kubernetes.io/projected/1947df5d-7c32-4906-9e5b-266da776ed8b-kube-api-access-tsmgm\") pod \"1947df5d-7c32-4906-9e5b-266da776ed8b\" (UID: \"1947df5d-7c32-4906-9e5b-266da776ed8b\") "
Nov 02 08:37:42.042506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.042356    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-home" pod "1947df5d-7c32-4906-9e5b-266da776ed8b" (UID: "1947df5d-7c32-4906-9e5b-266da776ed8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:42.042506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.042409    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-model-cache" pod "1947df5d-7c32-4906-9e5b-266da776ed8b" (UID: "1947df5d-7c32-4906-9e5b-266da776ed8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:42.042506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.042435    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-tmp-dir" pod "1947df5d-7c32-4906-9e5b-266da776ed8b" (UID: "1947df5d-7c32-4906-9e5b-266da776ed8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:42.042648 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.042579    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:42.042648 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.042593    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:42.042648 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.042603    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:42.044614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.044592    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-dshm" pod "1947df5d-7c32-4906-9e5b-266da776ed8b" (UID: "1947df5d-7c32-4906-9e5b-266da776ed8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:42.044614 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.044605    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1947df5d-7c32-4906-9e5b-266da776ed8b-tls-certs" pod "1947df5d-7c32-4906-9e5b-266da776ed8b" (UID: "1947df5d-7c32-4906-9e5b-266da776ed8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:42.044766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.044624    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1947df5d-7c32-4906-9e5b-266da776ed8b-kube-api-access-tsmgm" pod "1947df5d-7c32-4906-9e5b-266da776ed8b" (UID: "1947df5d-7c32-4906-9e5b-266da776ed8b"). InnerVolumeSpecName "kube-api-access-tsmgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:42.143792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.143760    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1947df5d-7c32-4906-9e5b-266da776ed8b-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:42.143792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.143789    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1947df5d-7c32-4906-9e5b-266da776ed8b-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:42.143792 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.143799    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsmgm\" (UniqueName: \"kubernetes.io/projected/1947df5d-7c32-4906-9e5b-266da776ed8b-kube-api-access-tsmgm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:42.320738 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.320698    4898 generic.go:356] "Generic (PLEG): container finished" podID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerID="93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98" exitCode=0
Nov 02 08:37:42.321299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.320741    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"
Nov 02 08:37:42.321299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.320790    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" event={"ID":"1947df5d-7c32-4906-9e5b-266da776ed8b","Type":"ContainerDied","Data":"93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98"}
Nov 02 08:37:42.321299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.320847    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j" event={"ID":"1947df5d-7c32-4906-9e5b-266da776ed8b","Type":"ContainerDied","Data":"39a54ec4990daf30fcc8149444d67876e80f6f9a4204ce2ab29d406a3815bee4"}
Nov 02 08:37:42.321299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.320873    4898 scope.go:122] "RemoveContainer" containerID="93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98"
Nov 02 08:37:42.322568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.322533    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g" event={"ID":"459a5dd3-a78c-4ba7-9338-e36624d2584c","Type":"ContainerStarted","Data":"2796bfb6e348788ecc3b4268bfdfc094782303027d4470b3bd798ad41890aeab"}
Nov 02 08:37:42.322669 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.322592    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g" event={"ID":"459a5dd3-a78c-4ba7-9338-e36624d2584c","Type":"ContainerStarted","Data":"16203a734e9693cd2e89fbe22804d662bc9fffb1f66e9d21edf6b5b6451ffb82"}
Nov 02 08:37:42.332138 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.332012    4898 scope.go:122] "RemoveContainer" containerID="93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98"
Nov 02 08:37:42.332389 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:42.332364    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98\": container with ID starting with 93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98 not found: ID does not exist" containerID="93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98"
Nov 02 08:37:42.332455 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.332401    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98"} err="failed to get container status \"93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98\": rpc error: code = NotFound desc = could not find container \"93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98\": container with ID starting with 93797bb66870cf35a49187c918e9143bb706d759baef4788934c1bc5da5eaf98 not found: ID does not exist"
Nov 02 08:37:42.392136 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.392094    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"]
Nov 02 08:37:42.395035 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.395002    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9db802/distinct-w4-9db802-kserve-5957f8db48-hfd6j"]
Nov 02 08:37:42.421380 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:42.421326    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g" podStartSLOduration=1.42131252 podStartE2EDuration="1.42131252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:37:42.41914704 +0000 UTC m=+4413.109474024" watchObservedRunningTime="2026-11-02 08:37:42.42131252 +0000 UTC m=+4413.111639483"
Nov 02 08:37:43.331625 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:43.331593    4898 generic.go:356] "Generic (PLEG): container finished" podID="459a5dd3-a78c-4ba7-9338-e36624d2584c" containerID="2796bfb6e348788ecc3b4268bfdfc094782303027d4470b3bd798ad41890aeab" exitCode=0
Nov 02 08:37:43.332044 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:43.331688    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g" event={"ID":"459a5dd3-a78c-4ba7-9338-e36624d2584c","Type":"ContainerDied","Data":"2796bfb6e348788ecc3b4268bfdfc094782303027d4470b3bd798ad41890aeab"}
Nov 02 08:37:43.464050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:43.464002    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" path="/var/lib/kubelet/pods/1947df5d-7c32-4906-9e5b-266da776ed8b/volumes"
Nov 02 08:37:44.334345 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.334317    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 08:37:44.338954 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.338928    4898 generic.go:356] "Generic (PLEG): container finished" podID="f9683de3-4678-406d-bd92-5cad8ae2637b" containerID="8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c" exitCode=0
Nov 02 08:37:44.339079 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.338967    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"
Nov 02 08:37:44.339079 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.339011    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" event={"ID":"f9683de3-4678-406d-bd92-5cad8ae2637b","Type":"ContainerDied","Data":"8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c"}
Nov 02 08:37:44.339207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.339073    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx" event={"ID":"f9683de3-4678-406d-bd92-5cad8ae2637b","Type":"ContainerDied","Data":"88c1aba8ec2e20821e43af3320a08a0cd68218b6c0901d576d4edc44b51c604c"}
Nov 02 08:37:44.339207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.339120    4898 scope.go:122] "RemoveContainer" containerID="8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c"
Nov 02 08:37:44.351007 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.350986    4898 scope.go:122] "RemoveContainer" containerID="8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c"
Nov 02 08:37:44.351284 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:44.351264    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c\": container with ID starting with 8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c not found: ID does not exist" containerID="8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c"
Nov 02 08:37:44.351377 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.351290    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c"} err="failed to get container status \"8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c\": rpc error: code = NotFound desc = could not find container \"8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c\": container with ID starting with 8a22f3bfdebcdb7c998eacf302d65c5c7e38af165b94d6b62b8e2343a8b6c99c not found: ID does not exist"
Nov 02 08:37:44.367346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.367321    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-token\" (UniqueName: \"kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-token\") pod \"f9683de3-4678-406d-bd92-5cad8ae2637b\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") "
Nov 02 08:37:44.367473 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.367388    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-kube-api-access-vd8j4\" (UniqueName: \"kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-kube-api-access-vd8j4\") pod \"f9683de3-4678-406d-bd92-5cad8ae2637b\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") "
Nov 02 08:37:44.367473 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.367459    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-socket\") pod \"f9683de3-4678-406d-bd92-5cad8ae2637b\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") "
Nov 02 08:37:44.367568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.367502    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-podinfo\") pod \"f9683de3-4678-406d-bd92-5cad8ae2637b\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") "
Nov 02 08:37:44.367568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.367555    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-data\") pod \"f9683de3-4678-406d-bd92-5cad8ae2637b\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") "
Nov 02 08:37:44.367658 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.367634    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9683de3-4678-406d-bd92-5cad8ae2637b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9683de3-4678-406d-bd92-5cad8ae2637b-istiod-ca-cert\") pod \"f9683de3-4678-406d-bd92-5cad8ae2637b\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") "
Nov 02 08:37:44.367718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.367666    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-envoy\") pod \"f9683de3-4678-406d-bd92-5cad8ae2637b\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") "
Nov 02 08:37:44.367718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.367702    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-certs\") pod \"f9683de3-4678-406d-bd92-5cad8ae2637b\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") "
Nov 02 08:37:44.367839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.367730    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-credential-socket\") pod \"f9683de3-4678-406d-bd92-5cad8ae2637b\" (UID: \"f9683de3-4678-406d-bd92-5cad8ae2637b\") "
Nov 02 08:37:44.367839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.367771    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-socket" pod "f9683de3-4678-406d-bd92-5cad8ae2637b" (UID: "f9683de3-4678-406d-bd92-5cad8ae2637b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:44.368007 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.367985    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-data" pod "f9683de3-4678-406d-bd92-5cad8ae2637b" (UID: "f9683de3-4678-406d-bd92-5cad8ae2637b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:44.368218 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.368195    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-credential-socket" pod "f9683de3-4678-406d-bd92-5cad8ae2637b" (UID: "f9683de3-4678-406d-bd92-5cad8ae2637b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:44.368304 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.368282    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.368304 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.368288    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-certs" pod "f9683de3-4678-406d-bd92-5cad8ae2637b" (UID: "f9683de3-4678-406d-bd92-5cad8ae2637b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:44.368412 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.368305    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.368469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.368447    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9683de3-4678-406d-bd92-5cad8ae2637b-istiod-ca-cert" pod "f9683de3-4678-406d-bd92-5cad8ae2637b" (UID: "f9683de3-4678-406d-bd92-5cad8ae2637b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:44.370013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.369984    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-token" pod "f9683de3-4678-406d-bd92-5cad8ae2637b" (UID: "f9683de3-4678-406d-bd92-5cad8ae2637b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:44.370013 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.370005    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-envoy" pod "f9683de3-4678-406d-bd92-5cad8ae2637b" (UID: "f9683de3-4678-406d-bd92-5cad8ae2637b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:37:44.370155 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.370104    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-kube-api-access-vd8j4" pod "f9683de3-4678-406d-bd92-5cad8ae2637b" (UID: "f9683de3-4678-406d-bd92-5cad8ae2637b"). InnerVolumeSpecName "kube-api-access-vd8j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:44.371314 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.370371    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-podinfo" pod "f9683de3-4678-406d-bd92-5cad8ae2637b" (UID: "f9683de3-4678-406d-bd92-5cad8ae2637b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:37:44.469372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.469343    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd8j4\" (UniqueName: \"kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-kube-api-access-vd8j4\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.469372 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.469370    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.469578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.469381    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9683de3-4678-406d-bd92-5cad8ae2637b-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.469578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.469391    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.469578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.469399    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.469578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.469407    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9683de3-4678-406d-bd92-5cad8ae2637b-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.469578 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.469414    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9683de3-4678-406d-bd92-5cad8ae2637b-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.485567 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.485540    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:44.570496 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.570468    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-openshift-ca\") pod \"459a5dd3-a78c-4ba7-9338-e36624d2584c\" (UID: \"459a5dd3-a78c-4ba7-9338-e36624d2584c\") "
Nov 02 08:37:44.570666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.570525    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459a5dd3-a78c-4ba7-9338-e36624d2584c-kube-api-access-9xgxq\" (UniqueName: \"kubernetes.io/projected/459a5dd3-a78c-4ba7-9338-e36624d2584c-kube-api-access-9xgxq\") pod \"459a5dd3-a78c-4ba7-9338-e36624d2584c\" (UID: \"459a5dd3-a78c-4ba7-9338-e36624d2584c\") "
Nov 02 08:37:44.570666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.570590    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-xks-ca\") pod \"459a5dd3-a78c-4ba7-9338-e36624d2584c\" (UID: \"459a5dd3-a78c-4ba7-9338-e36624d2584c\") "
Nov 02 08:37:44.570817 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.570789    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-openshift-ca" pod "459a5dd3-a78c-4ba7-9338-e36624d2584c" (UID: "459a5dd3-a78c-4ba7-9338-e36624d2584c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:44.570991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.570967    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.572843 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.572816    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-xks-ca" pod "459a5dd3-a78c-4ba7-9338-e36624d2584c" (UID: "459a5dd3-a78c-4ba7-9338-e36624d2584c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:37:44.572967 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.572853    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459a5dd3-a78c-4ba7-9338-e36624d2584c-kube-api-access-9xgxq" pod "459a5dd3-a78c-4ba7-9338-e36624d2584c" (UID: "459a5dd3-a78c-4ba7-9338-e36624d2584c"). InnerVolumeSpecName "kube-api-access-9xgxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:44.671989 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.671956    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xgxq\" (UniqueName: \"kubernetes.io/projected/459a5dd3-a78c-4ba7-9338-e36624d2584c-kube-api-access-9xgxq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.671989 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.671983    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a5dd3-a78c-4ba7-9338-e36624d2584c-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:44.685944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.685910    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"]
Nov 02 08:37:44.694072 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:44.694047    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-354660-openshift-default-79d5658cf-5qnjx"]
Nov 02 08:37:44.988577 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:44.988533    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:37:45.352359 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:45.352260    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g" event={"ID":"459a5dd3-a78c-4ba7-9338-e36624d2584c","Type":"ContainerDied","Data":"16203a734e9693cd2e89fbe22804d662bc9fffb1f66e9d21edf6b5b6451ffb82"}
Nov 02 08:37:45.352859 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:45.352372    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16203a734e9693cd2e89fbe22804d662bc9fffb1f66e9d21edf6b5b6451ffb82"
Nov 02 08:37:45.352859 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:45.352473    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"
Nov 02 08:37:45.464080 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:45.464043    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9683de3-4678-406d-bd92-5cad8ae2637b" path="/var/lib/kubelet/pods/f9683de3-4678-406d-bd92-5cad8ae2637b/volumes"
Nov 02 08:37:45.937786 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:45.937740    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"]
Nov 02 08:37:45.948039 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:45.948009    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"]
Nov 02 08:37:48.433899 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:48.433846    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv" podUID="c3566a46-846d-46a7-86e5-26159d70d0dc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:48.434383 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:48.433851    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z" podUID="8f1541ea-41ea-4c23-8f9c-9279f17592ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:50.438785 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:50.438742    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4" podUID="eea92ad5-8888-44e8-b546-a58d39a3d37e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:50.439235 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:50.438810    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99" podUID="d031a556-bb96-4947-bdc9-a96b70bac427" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:37:56.157173 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.157144    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"
Nov 02 08:37:56.213250 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.213222    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"
Nov 02 08:37:56.294322 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.294290    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f1541ea-41ea-4c23-8f9c-9279f17592ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f1541ea-41ea-4c23-8f9c-9279f17592ca-plugins-config-volume\") pod \"8f1541ea-41ea-4c23-8f9c-9279f17592ca\" (UID: \"8f1541ea-41ea-4c23-8f9c-9279f17592ca\") "
Nov 02 08:37:56.294507 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.294343    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f1541ea-41ea-4c23-8f9c-9279f17592ca-kube-api-access-rwz98\" (UniqueName: \"kubernetes.io/projected/8f1541ea-41ea-4c23-8f9c-9279f17592ca-kube-api-access-rwz98\") pod \"8f1541ea-41ea-4c23-8f9c-9279f17592ca\" (UID: \"8f1541ea-41ea-4c23-8f9c-9279f17592ca\") "
Nov 02 08:37:56.294507 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.294449    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3566a46-846d-46a7-86e5-26159d70d0dc-kube-api-access-4q526\" (UniqueName: \"kubernetes.io/projected/c3566a46-846d-46a7-86e5-26159d70d0dc-kube-api-access-4q526\") pod \"c3566a46-846d-46a7-86e5-26159d70d0dc\" (UID: \"c3566a46-846d-46a7-86e5-26159d70d0dc\") "
Nov 02 08:37:56.294507 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.294491    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3566a46-846d-46a7-86e5-26159d70d0dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3566a46-846d-46a7-86e5-26159d70d0dc-plugins-config-volume\") pod \"c3566a46-846d-46a7-86e5-26159d70d0dc\" (UID: \"c3566a46-846d-46a7-86e5-26159d70d0dc\") "
Nov 02 08:37:56.294807 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.294776    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f1541ea-41ea-4c23-8f9c-9279f17592ca-plugins-config-volume" pod "8f1541ea-41ea-4c23-8f9c-9279f17592ca" (UID: "8f1541ea-41ea-4c23-8f9c-9279f17592ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:56.294997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.294909    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f1541ea-41ea-4c23-8f9c-9279f17592ca-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:56.294997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.294957    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3566a46-846d-46a7-86e5-26159d70d0dc-plugins-config-volume" pod "c3566a46-846d-46a7-86e5-26159d70d0dc" (UID: "c3566a46-846d-46a7-86e5-26159d70d0dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:37:56.296626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.296607    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3566a46-846d-46a7-86e5-26159d70d0dc-kube-api-access-4q526" pod "c3566a46-846d-46a7-86e5-26159d70d0dc" (UID: "c3566a46-846d-46a7-86e5-26159d70d0dc"). InnerVolumeSpecName "kube-api-access-4q526". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:56.296988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.296958    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1541ea-41ea-4c23-8f9c-9279f17592ca-kube-api-access-rwz98" pod "8f1541ea-41ea-4c23-8f9c-9279f17592ca" (UID: "8f1541ea-41ea-4c23-8f9c-9279f17592ca"). InnerVolumeSpecName "kube-api-access-rwz98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:37:56.395532 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.395448    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q526\" (UniqueName: \"kubernetes.io/projected/c3566a46-846d-46a7-86e5-26159d70d0dc-kube-api-access-4q526\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:56.395532 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.395478    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3566a46-846d-46a7-86e5-26159d70d0dc-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:56.395532 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.395493    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwz98\" (UniqueName: \"kubernetes.io/projected/8f1541ea-41ea-4c23-8f9c-9279f17592ca-kube-api-access-rwz98\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:37:56.451988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.451955    4898 generic.go:356] "Generic (PLEG): container finished" podID="8f1541ea-41ea-4c23-8f9c-9279f17592ca" containerID="2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9" exitCode=0
Nov 02 08:37:56.452194 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.452001    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"
Nov 02 08:37:56.452194 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.452042    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z" event={"ID":"8f1541ea-41ea-4c23-8f9c-9279f17592ca","Type":"ContainerDied","Data":"2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9"}
Nov 02 08:37:56.452194 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.452094    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z" event={"ID":"8f1541ea-41ea-4c23-8f9c-9279f17592ca","Type":"ContainerDied","Data":"685f46fd75a23c11e3ffd4a0a989a6d2ed0f3a944773d2fb49878e7cd4162a56"}
Nov 02 08:37:56.452194 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.452122    4898 scope.go:122] "RemoveContainer" containerID="2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9"
Nov 02 08:37:56.453511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.453489    4898 generic.go:356] "Generic (PLEG): container finished" podID="c3566a46-846d-46a7-86e5-26159d70d0dc" containerID="8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e" exitCode=0
Nov 02 08:37:56.453609 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.453594    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv" event={"ID":"c3566a46-846d-46a7-86e5-26159d70d0dc","Type":"ContainerDied","Data":"8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e"}
Nov 02 08:37:56.453673 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.453627    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv" event={"ID":"c3566a46-846d-46a7-86e5-26159d70d0dc","Type":"ContainerDied","Data":"89a5199c23a78069bd7051c24d9c8d89a16843fa84d6fe8e3923dfb99f1b2fcf"}
Nov 02 08:37:56.453739 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.453689    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"
Nov 02 08:37:56.463506 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.463486    4898 scope.go:122] "RemoveContainer" containerID="2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9"
Nov 02 08:37:56.463833 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:56.463812    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9\": container with ID starting with 2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9 not found: ID does not exist" containerID="2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9"
Nov 02 08:37:56.463945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.463840    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9"} err="failed to get container status \"2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9\": rpc error: code = NotFound desc = could not find container \"2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9\": container with ID starting with 2d6bd966ee4ee08a0eb444a717b321be3503393ce468e962cf4120c1fcbc3fa9 not found: ID does not exist"
Nov 02 08:37:56.463945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.463859    4898 scope.go:122] "RemoveContainer" containerID="8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e"
Nov 02 08:37:56.476025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.476000    4898 scope.go:122] "RemoveContainer" containerID="8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e"
Nov 02 08:37:56.476324 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:37:56.476300    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e\": container with ID starting with 8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e not found: ID does not exist" containerID="8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e"
Nov 02 08:37:56.476392 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.476336    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e"} err="failed to get container status \"8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e\": rpc error: code = NotFound desc = could not find container \"8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e\": container with ID starting with 8d200ef31dfc2d8b1d016dc008f505f3c70a0cfac19da6677acdf6ab09adff6e not found: ID does not exist"
Nov 02 08:37:56.506401 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.506379    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"]
Nov 02 08:37:56.510911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.510875    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-71a41c-5dcfbbccbc-q559z"]
Nov 02 08:37:56.544536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.544514    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"]
Nov 02 08:37:56.550206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:56.550183    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-71a41c-64d6f8f7bc-mf5fv"]
Nov 02 08:37:57.463902 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:57.463854    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f1541ea-41ea-4c23-8f9c-9279f17592ca" path="/var/lib/kubelet/pods/8f1541ea-41ea-4c23-8f9c-9279f17592ca/volumes"
Nov 02 08:37:57.464327 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:57.464311    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3566a46-846d-46a7-86e5-26159d70d0dc" path="/var/lib/kubelet/pods/c3566a46-846d-46a7-86e5-26159d70d0dc/volumes"
Nov 02 08:37:59.950562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:37:59.950532    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"
Nov 02 08:38:00.039236 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.039204    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea92ad5-8888-44e8-b546-a58d39a3d37e-kube-api-access-2q9xh\" (UniqueName: \"kubernetes.io/projected/eea92ad5-8888-44e8-b546-a58d39a3d37e-kube-api-access-2q9xh\") pod \"eea92ad5-8888-44e8-b546-a58d39a3d37e\" (UID: \"eea92ad5-8888-44e8-b546-a58d39a3d37e\") "
Nov 02 08:38:00.039411 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.039258    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea92ad5-8888-44e8-b546-a58d39a3d37e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea92ad5-8888-44e8-b546-a58d39a3d37e-plugins-config-volume\") pod \"eea92ad5-8888-44e8-b546-a58d39a3d37e\" (UID: \"eea92ad5-8888-44e8-b546-a58d39a3d37e\") "
Nov 02 08:38:00.039688 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.039658    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea92ad5-8888-44e8-b546-a58d39a3d37e-plugins-config-volume" pod "eea92ad5-8888-44e8-b546-a58d39a3d37e" (UID: "eea92ad5-8888-44e8-b546-a58d39a3d37e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:38:00.041401 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.041376    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea92ad5-8888-44e8-b546-a58d39a3d37e-kube-api-access-2q9xh" pod "eea92ad5-8888-44e8-b546-a58d39a3d37e" (UID: "eea92ad5-8888-44e8-b546-a58d39a3d37e"). InnerVolumeSpecName "kube-api-access-2q9xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:00.140944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.140865    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q9xh\" (UniqueName: \"kubernetes.io/projected/eea92ad5-8888-44e8-b546-a58d39a3d37e-kube-api-access-2q9xh\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:00.140944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.140928    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea92ad5-8888-44e8-b546-a58d39a3d37e-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:00.506579 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.506536    4898 generic.go:356] "Generic (PLEG): container finished" podID="d031a556-bb96-4947-bdc9-a96b70bac427" containerID="d7229b4c958a98d898db1c50c8554295b82abbe01b1271e6f6f9f9c523c9bd1b" exitCode=0
Nov 02 08:38:00.506766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.506605    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99" event={"ID":"d031a556-bb96-4947-bdc9-a96b70bac427","Type":"ContainerDied","Data":"d7229b4c958a98d898db1c50c8554295b82abbe01b1271e6f6f9f9c523c9bd1b"}
Nov 02 08:38:00.508650 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.508621    4898 generic.go:356] "Generic (PLEG): container finished" podID="eea92ad5-8888-44e8-b546-a58d39a3d37e" containerID="e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b" exitCode=0
Nov 02 08:38:00.508791 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.508676    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"
Nov 02 08:38:00.508867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.508847    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4" event={"ID":"eea92ad5-8888-44e8-b546-a58d39a3d37e","Type":"ContainerDied","Data":"e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b"}
Nov 02 08:38:00.508946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.508914    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4" event={"ID":"eea92ad5-8888-44e8-b546-a58d39a3d37e","Type":"ContainerDied","Data":"5ff666df3b82f8e8b9b039dfc562f7981f4e0ef1c1dfbadf4e6f413da8ded609"}
Nov 02 08:38:00.509016 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.508948    4898 scope.go:122] "RemoveContainer" containerID="e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b"
Nov 02 08:38:00.521938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.521914    4898 scope.go:122] "RemoveContainer" containerID="e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b"
Nov 02 08:38:00.522382 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:38:00.522274    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b\": container with ID starting with e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b not found: ID does not exist" containerID="e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b"
Nov 02 08:38:00.522382 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.522308    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b"} err="failed to get container status \"e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b\": rpc error: code = NotFound desc = could not find container \"e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b\": container with ID starting with e649f265a5540aeabea8958fdb3e3886eb45180d12bab34af5cf97fe5abb9e1b not found: ID does not exist"
Nov 02 08:38:00.528607 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.528588    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 08:38:00.563638 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.563606    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"]
Nov 02 08:38:00.575681 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.575651    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9db802-744b4d58c-r6rw4"]
Nov 02 08:38:00.654623 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.654590    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d031a556-bb96-4947-bdc9-a96b70bac427-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d031a556-bb96-4947-bdc9-a96b70bac427-plugins-config-volume\") pod \"d031a556-bb96-4947-bdc9-a96b70bac427\" (UID: \"d031a556-bb96-4947-bdc9-a96b70bac427\") "
Nov 02 08:38:00.654805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.654635    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d031a556-bb96-4947-bdc9-a96b70bac427-kube-api-access-8c7kt\" (UniqueName: \"kubernetes.io/projected/d031a556-bb96-4947-bdc9-a96b70bac427-kube-api-access-8c7kt\") pod \"d031a556-bb96-4947-bdc9-a96b70bac427\" (UID: \"d031a556-bb96-4947-bdc9-a96b70bac427\") "
Nov 02 08:38:00.655050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.655021    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d031a556-bb96-4947-bdc9-a96b70bac427-plugins-config-volume" pod "d031a556-bb96-4947-bdc9-a96b70bac427" (UID: "d031a556-bb96-4947-bdc9-a96b70bac427"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:38:00.656881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.656851    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d031a556-bb96-4947-bdc9-a96b70bac427-kube-api-access-8c7kt" pod "d031a556-bb96-4947-bdc9-a96b70bac427" (UID: "d031a556-bb96-4947-bdc9-a96b70bac427"). InnerVolumeSpecName "kube-api-access-8c7kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:00.756259 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.756222    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d031a556-bb96-4947-bdc9-a96b70bac427-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:00.756259 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.756256    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c7kt\" (UniqueName: \"kubernetes.io/projected/d031a556-bb96-4947-bdc9-a96b70bac427-kube-api-access-8c7kt\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:00.960366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.960319    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main" containerID="cri-o://447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490" gracePeriod=45
Nov 02 08:38:00.968437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:00.968401    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main" containerID="cri-o://cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788" gracePeriod=45
Nov 02 08:38:01.245275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.245243    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 08:38:01.266969 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.266941    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 08:38:01.362770 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.362736    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-home\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-home\") pod \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") "
Nov 02 08:38:01.362995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.362807    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c3003c-49e7-44cb-abb4-06ddddcbfafe-kube-api-access-ts6jr\" (UniqueName: \"kubernetes.io/projected/12c3003c-49e7-44cb-abb4-06ddddcbfafe-kube-api-access-ts6jr\") pod \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") "
Nov 02 08:38:01.362995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.362839    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tls-certs\" (UniqueName: \"kubernetes.io/secret/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tls-certs\") pod \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") "
Nov 02 08:38:01.362995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.362919    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-model-cache\") pod \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") "
Nov 02 08:38:01.362995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.362959    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tls-certs\") pod \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") "
Nov 02 08:38:01.362995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.362989    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tmp-dir\") pod \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") "
Nov 02 08:38:01.363295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.363029    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-kube-api-access-xn847\" (UniqueName: \"kubernetes.io/projected/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-kube-api-access-xn847\") pod \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") "
Nov 02 08:38:01.363295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.363065    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-dshm\") pod \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") "
Nov 02 08:38:01.363295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.363134    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-home" pod "9c5688fd-298b-49d4-b852-4c4f6d4f40d4" (UID: "9c5688fd-298b-49d4-b852-4c4f6d4f40d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:01.363295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.363213    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-dshm\") pod \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") "
Nov 02 08:38:01.363295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.363254    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-model-cache\") pod \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") "
Nov 02 08:38:01.363598 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.363326    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-home\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-home\") pod \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\" (UID: \"12c3003c-49e7-44cb-abb4-06ddddcbfafe\") "
Nov 02 08:38:01.363598 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.363360    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tmp-dir\") pod \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\" (UID: \"9c5688fd-298b-49d4-b852-4c4f6d4f40d4\") "
Nov 02 08:38:01.363784 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.363764    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:01.364083 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.364059    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tmp-dir" pod "9c5688fd-298b-49d4-b852-4c4f6d4f40d4" (UID: "9c5688fd-298b-49d4-b852-4c4f6d4f40d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:01.365344 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.365044    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tmp-dir" pod "12c3003c-49e7-44cb-abb4-06ddddcbfafe" (UID: "12c3003c-49e7-44cb-abb4-06ddddcbfafe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:01.365585 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.365526    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-model-cache" pod "12c3003c-49e7-44cb-abb4-06ddddcbfafe" (UID: "12c3003c-49e7-44cb-abb4-06ddddcbfafe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:01.365691 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.365632    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-dshm" pod "9c5688fd-298b-49d4-b852-4c4f6d4f40d4" (UID: "9c5688fd-298b-49d4-b852-4c4f6d4f40d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:01.365758 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.365694    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-model-cache" pod "9c5688fd-298b-49d4-b852-4c4f6d4f40d4" (UID: "9c5688fd-298b-49d4-b852-4c4f6d4f40d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:01.365758 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.365711    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c3003c-49e7-44cb-abb4-06ddddcbfafe-kube-api-access-ts6jr" pod "12c3003c-49e7-44cb-abb4-06ddddcbfafe" (UID: "12c3003c-49e7-44cb-abb4-06ddddcbfafe"). InnerVolumeSpecName "kube-api-access-ts6jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:01.365758 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.365747    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-home" pod "12c3003c-49e7-44cb-abb4-06ddddcbfafe" (UID: "12c3003c-49e7-44cb-abb4-06ddddcbfafe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:01.365970 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.365947    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tls-certs" pod "9c5688fd-298b-49d4-b852-4c4f6d4f40d4" (UID: "9c5688fd-298b-49d4-b852-4c4f6d4f40d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:38:01.366736 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.366714    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-dshm" pod "12c3003c-49e7-44cb-abb4-06ddddcbfafe" (UID: "12c3003c-49e7-44cb-abb4-06ddddcbfafe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:01.367454 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.367426    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tls-certs" pod "12c3003c-49e7-44cb-abb4-06ddddcbfafe" (UID: "12c3003c-49e7-44cb-abb4-06ddddcbfafe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:38:01.368207 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.368181    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-kube-api-access-xn847" pod "9c5688fd-298b-49d4-b852-4c4f6d4f40d4" (UID: "9c5688fd-298b-49d4-b852-4c4f6d4f40d4"). InnerVolumeSpecName "kube-api-access-xn847". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:01.421638 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.421597    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"]
Nov 02 08:38:01.422282 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422264    4898 cpu_manager.go:414] "containerMap: removing container" podUID="eea92ad5-8888-44e8-b546-a58d39a3d37e" containerName="payload-pre-processing"
Nov 02 08:38:01.422339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422285    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea92ad5-8888-44e8-b546-a58d39a3d37e" containerName="payload-pre-processing"
Nov 02 08:38:01.422339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422306    4898 cpu_manager.go:414] "containerMap: removing container" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main"
Nov 02 08:38:01.422339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422313    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main"
Nov 02 08:38:01.422339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422330    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f9683de3-4678-406d-bd92-5cad8ae2637b" containerName="istio-proxy"
Nov 02 08:38:01.422339 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422337    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9683de3-4678-406d-bd92-5cad8ae2637b" containerName="istio-proxy"
Nov 02 08:38:01.422499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422350    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main"
Nov 02 08:38:01.422499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422358    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main"
Nov 02 08:38:01.422499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422369    4898 cpu_manager.go:414] "containerMap: removing container" podUID="459a5dd3-a78c-4ba7-9338-e36624d2584c" containerName="revoke-keys"
Nov 02 08:38:01.422499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422376    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459a5dd3-a78c-4ba7-9338-e36624d2584c" containerName="revoke-keys"
Nov 02 08:38:01.422499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422392    4898 cpu_manager.go:414] "containerMap: removing container" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main"
Nov 02 08:38:01.422499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422399    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main"
Nov 02 08:38:01.422499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422415    4898 cpu_manager.go:414] "containerMap: removing container" podUID="c3566a46-846d-46a7-86e5-26159d70d0dc" containerName="payload-processing"
Nov 02 08:38:01.422499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422423    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3566a46-846d-46a7-86e5-26159d70d0dc" containerName="payload-processing"
Nov 02 08:38:01.422499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422437    4898 cpu_manager.go:414] "containerMap: removing container" podUID="8f1541ea-41ea-4c23-8f9c-9279f17592ca" containerName="payload-pre-processing"
Nov 02 08:38:01.422499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422444    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f1541ea-41ea-4c23-8f9c-9279f17592ca" containerName="payload-pre-processing"
Nov 02 08:38:01.422499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422462    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d031a556-bb96-4947-bdc9-a96b70bac427" containerName="payload-processing"
Nov 02 08:38:01.422499 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422470    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d031a556-bb96-4947-bdc9-a96b70bac427" containerName="payload-processing"
Nov 02 08:38:01.422919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422552    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="459a5dd3-a78c-4ba7-9338-e36624d2584c" containerName="revoke-keys"
Nov 02 08:38:01.422919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422564    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerName="main"
Nov 02 08:38:01.422919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422574    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3566a46-846d-46a7-86e5-26159d70d0dc" containerName="payload-processing"
Nov 02 08:38:01.422919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422586    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f1541ea-41ea-4c23-8f9c-9279f17592ca" containerName="payload-pre-processing"
Nov 02 08:38:01.422919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422595    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d031a556-bb96-4947-bdc9-a96b70bac427" containerName="payload-processing"
Nov 02 08:38:01.422919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422606    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea92ad5-8888-44e8-b546-a58d39a3d37e" containerName="payload-pre-processing"
Nov 02 08:38:01.422919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422615    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerName="main"
Nov 02 08:38:01.422919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422625    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9683de3-4678-406d-bd92-5cad8ae2637b" containerName="istio-proxy"
Nov 02 08:38:01.422919 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.422633    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1947df5d-7c32-4906-9e5b-266da776ed8b" containerName="main"
Nov 02 08:38:01.427676 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.427650    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.437934 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.437881    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99" podUID="d031a556-bb96-4947-bdc9-a96b70bac427" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.70:50052\" within 1s: context deadline exceeded"
Nov 02 08:38:01.438071 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:38:01.437980    4898 logging.go:55] [core] [Channel #18157 SubChannel #18158]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.70:50052", ServerName: "10.132.0.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.70:50052: operation was canceled"
Nov 02 08:38:01.438071 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.437982    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 08:38:01.445929 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.445790    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"]
Nov 02 08:38:01.464634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.464554    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:01.464634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.464593    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn847\" (UniqueName: \"kubernetes.io/projected/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-kube-api-access-xn847\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:01.464634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.464610    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:01.464634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.464623    4898 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-dshm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:01.464634 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.464636    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:01.464990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.464649    4898 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c3003c-49e7-44cb-abb4-06ddddcbfafe-home\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:01.464990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.464661    4898 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tmp-dir\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:01.464990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.464674    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts6jr\" (UniqueName: \"kubernetes.io/projected/12c3003c-49e7-44cb-abb4-06ddddcbfafe-kube-api-access-ts6jr\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:01.464990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.464687    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c3003c-49e7-44cb-abb4-06ddddcbfafe-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:01.464990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.464701    4898 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-model-cache\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:01.464990 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.464713    4898 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5688fd-298b-49d4-b852-4c4f6d4f40d4-tls-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:01.465337 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.465304    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea92ad5-8888-44e8-b546-a58d39a3d37e" path="/var/lib/kubelet/pods/eea92ad5-8888-44e8-b546-a58d39a3d37e/volumes"
Nov 02 08:38:01.524160 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.524121    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9db802-openshift-default-dockercfg-4p9sf\""
Nov 02 08:38:01.526738 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.526714    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"
Nov 02 08:38:01.526910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.526738    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99" event={"ID":"d031a556-bb96-4947-bdc9-a96b70bac427","Type":"ContainerDied","Data":"c3bb54361b7837be0c43befc2d93d013bac1029cd6c3363b0c0145fb5126d2d6"}
Nov 02 08:38:01.526910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.526835    4898 scope.go:122] "RemoveContainer" containerID="d7229b4c958a98d898db1c50c8554295b82abbe01b1271e6f6f9f9c523c9bd1b"
Nov 02 08:38:01.529705 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.529681    4898 generic.go:356] "Generic (PLEG): container finished" podID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" containerID="cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788" exitCode=0
Nov 02 08:38:01.529831 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.529731    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"
Nov 02 08:38:01.529831 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.529752    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" event={"ID":"9c5688fd-298b-49d4-b852-4c4f6d4f40d4","Type":"ContainerDied","Data":"cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788"}
Nov 02 08:38:01.529831 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.529799    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb" event={"ID":"9c5688fd-298b-49d4-b852-4c4f6d4f40d4","Type":"ContainerDied","Data":"5abaf14488d2eac556d03792e7c8a206e9a3a006db7bb2990eeabae87916efbe"}
Nov 02 08:38:01.531293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.531268    4898 generic.go:356] "Generic (PLEG): container finished" podID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" containerID="447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490" exitCode=0
Nov 02 08:38:01.531388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.531316    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"
Nov 02 08:38:01.531388 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.531363    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" event={"ID":"12c3003c-49e7-44cb-abb4-06ddddcbfafe","Type":"ContainerDied","Data":"447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490"}
Nov 02 08:38:01.531484 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.531389    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq" event={"ID":"12c3003c-49e7-44cb-abb4-06ddddcbfafe","Type":"ContainerDied","Data":"81598b8dcb6ddc54d36e9dd1a47f5f285449a9996bc2538390cfe472fc82234b"}
Nov 02 08:38:01.538025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.538002    4898 scope.go:122] "RemoveContainer" containerID="cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788"
Nov 02 08:38:01.546755 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.546727    4898 scope.go:122] "RemoveContainer" containerID="cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788"
Nov 02 08:38:01.547080 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:38:01.547054    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788\": container with ID starting with cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788 not found: ID does not exist" containerID="cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788"
Nov 02 08:38:01.547132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.547087    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788"} err="failed to get container status \"cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788\": rpc error: code = NotFound desc = could not find container \"cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788\": container with ID starting with cefbacd60ef0486cf685d40d727a2f7c0394840293366f25e0085ac9b6661788 not found: ID does not exist"
Nov 02 08:38:01.547132 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.547103    4898 scope.go:122] "RemoveContainer" containerID="447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490"
Nov 02 08:38:01.555666 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.555648    4898 scope.go:122] "RemoveContainer" containerID="447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490"
Nov 02 08:38:01.555946 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:38:01.555927    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490\": container with ID starting with 447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490 not found: ID does not exist" containerID="447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490"
Nov 02 08:38:01.556011 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.555950    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490"} err="failed to get container status \"447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490\": rpc error: code = NotFound desc = could not find container \"447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490\": container with ID starting with 447bb6250714a00e0ff9d5ea6cd7d6b8e1dc0a84a34ccf46f2740a9cdd7e3490 not found: ID does not exist"
Nov 02 08:38:01.565760 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.565739    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrf9k\" (UniqueName: \"kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-kube-api-access-rrf9k\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.565866 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.565774    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.565866 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.565807    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.565982 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.565957    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.566033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.565992    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.566033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.566010    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.566146 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.566080    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.566227 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.566212    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.566289 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.566256    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.579046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.579014    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"]
Nov 02 08:38:01.583167 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.583143    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9db802-544bb54dbf-jvw99"]
Nov 02 08:38:01.616516 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.616479    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"]
Nov 02 08:38:01.620364 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.620338    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-337764/distinct-w2-337764-kserve-67cf9d8469-xhbsq"]
Nov 02 08:38:01.653877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.653843    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"]
Nov 02 08:38:01.657787 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.657759    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-337764/simulator-w2-337764-kserve-64d7c65d99-t6fvb"]
Nov 02 08:38:01.667437 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.667414    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.667543 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.667457    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.667543 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.667512    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.667657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.667545    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.667657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.667570    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.667657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.667607    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.667657 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.667642    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.667945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.667677    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.667945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.667724    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrf9k\" (UniqueName: \"kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-kube-api-access-rrf9k\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.668064 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.667949    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-data\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.668064 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.668043    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-credential-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.668189 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.668111    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-certs\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.668257 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.668237    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-socket\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.668321 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.668291    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istiod-ca-cert\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.669994 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.669969    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-envoy\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.670318 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.670300    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-podinfo\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.691033 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.691003    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-token\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.691131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.691037    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrf9k\" (UniqueName: \"kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-kube-api-access-rrf9k\") pod \"e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") " pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:01.758945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:01.758852    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:02.076288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:02.076253    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"]
Nov 02 08:38:02.077321 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:38:02.077290    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2ea646_5f4a_4387_8f2c_84fbca7b2089.slice/crio-bd8d9e081e67b358568b32b46452aabf1fa11e5a8aab1d2b2763c938ee6110b1 WatchSource:0}: Error finding container bd8d9e081e67b358568b32b46452aabf1fa11e5a8aab1d2b2763c938ee6110b1: Status 404 returned error can't find the container with id bd8d9e081e67b358568b32b46452aabf1fa11e5a8aab1d2b2763c938ee6110b1
Nov 02 08:38:02.079542 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:02.079503    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:02.079629 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:02.079593    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:02.079682 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:02.079635    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:38:02.541446 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:02.541407    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k" event={"ID":"7e2ea646-5f4a-4387-8f2c-84fbca7b2089","Type":"ContainerStarted","Data":"b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123"}
Nov 02 08:38:02.541650 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:02.541471    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k" event={"ID":"7e2ea646-5f4a-4387-8f2c-84fbca7b2089","Type":"ContainerStarted","Data":"bd8d9e081e67b358568b32b46452aabf1fa11e5a8aab1d2b2763c938ee6110b1"}
Nov 02 08:38:02.614105 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:02.614052    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k" podStartSLOduration=1.614037822 podStartE2EDuration="1.614037822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:38:02.611663448 +0000 UTC m=+4433.301990439" watchObservedRunningTime="2026-11-02 08:38:02.614037822 +0000 UTC m=+4433.304364785"
Nov 02 08:38:02.772914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:02.760985    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:03.538669 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:03.538622    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c3003c-49e7-44cb-abb4-06ddddcbfafe" path="/var/lib/kubelet/pods/12c3003c-49e7-44cb-abb4-06ddddcbfafe/volumes"
Nov 02 08:38:03.539275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:03.539253    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c5688fd-298b-49d4-b852-4c4f6d4f40d4" path="/var/lib/kubelet/pods/9c5688fd-298b-49d4-b852-4c4f6d4f40d4/volumes"
Nov 02 08:38:03.546412 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:03.539720    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d031a556-bb96-4947-bdc9-a96b70bac427" path="/var/lib/kubelet/pods/d031a556-bb96-4947-bdc9-a96b70bac427/volumes"
Nov 02 08:38:03.775913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:03.768995    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.166:15021/healthz/ready\": dial tcp 10.132.1.166:15021: connect: connection refused" start-of-body=
Nov 02 08:38:03.775913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:03.769070    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k" podUID="7e2ea646-5f4a-4387-8f2c-84fbca7b2089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.166:15021/healthz/ready\": dial tcp 10.132.1.166:15021: connect: connection refused"
Nov 02 08:38:04.760914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:04.759247    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.166:15021/healthz/ready\": dial tcp 10.132.1.166:15021: connect: connection refused" start-of-body=
Nov 02 08:38:04.760914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:04.759323    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k" podUID="7e2ea646-5f4a-4387-8f2c-84fbca7b2089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.166:15021/healthz/ready\": dial tcp 10.132.1.166:15021: connect: connection refused"
Nov 02 08:38:05.762284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:05.762247    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.166:15021/healthz/ready\": dial tcp 10.132.1.166:15021: connect: connection refused" start-of-body=
Nov 02 08:38:05.762866 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:05.762834    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k" podUID="7e2ea646-5f4a-4387-8f2c-84fbca7b2089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.166:15021/healthz/ready\": dial tcp 10.132.1.166:15021: connect: connection refused"
Nov 02 08:38:06.760913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:06.760270    4898 patch_prober.go:29] interesting pod/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.166:15021/healthz/ready\": dial tcp 10.132.1.166:15021: connect: connection refused" start-of-body=
Nov 02 08:38:06.760913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:06.760348    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k" podUID="7e2ea646-5f4a-4387-8f2c-84fbca7b2089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.166:15021/healthz/ready\": dial tcp 10.132.1.166:15021: connect: connection refused"
Nov 02 08:38:06.997737 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:06.997695    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"]
Nov 02 08:38:06.999664 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:06.999039    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k" podUID="7e2ea646-5f4a-4387-8f2c-84fbca7b2089" containerName="istio-proxy" containerID="cri-o://b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123" gracePeriod=30
Nov 02 08:38:13.660765 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.660735    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:13.676626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.676564    4898 generic.go:356] "Generic (PLEG): container finished" podID="7e2ea646-5f4a-4387-8f2c-84fbca7b2089" containerID="b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123" exitCode=0
Nov 02 08:38:13.676626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.676611    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"
Nov 02 08:38:13.676781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.676648    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k" event={"ID":"7e2ea646-5f4a-4387-8f2c-84fbca7b2089","Type":"ContainerDied","Data":"b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123"}
Nov 02 08:38:13.676781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.676722    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k" event={"ID":"7e2ea646-5f4a-4387-8f2c-84fbca7b2089","Type":"ContainerDied","Data":"bd8d9e081e67b358568b32b46452aabf1fa11e5a8aab1d2b2763c938ee6110b1"}
Nov 02 08:38:13.676781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.676754    4898 scope.go:122] "RemoveContainer" containerID="b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123"
Nov 02 08:38:13.686537 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.686519    4898 scope.go:122] "RemoveContainer" containerID="b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123"
Nov 02 08:38:13.686820 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:38:13.686794    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123\": container with ID starting with b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123 not found: ID does not exist" containerID="b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123"
Nov 02 08:38:13.686907 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.686830    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123"} err="failed to get container status \"b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123\": rpc error: code = NotFound desc = could not find container \"b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123\": container with ID starting with b0f2b4f8effe4d632acefb9e0e4b63d17d17e8faf0ee8f60a2fd314f80afa123 not found: ID does not exist"
Nov 02 08:38:13.737120 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737082    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-certs\") pod \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") "
Nov 02 08:38:13.737312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737143    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-credential-socket\") pod \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") "
Nov 02 08:38:13.737312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737179    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-socket\") pod \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") "
Nov 02 08:38:13.737312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737208    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-envoy\") pod \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") "
Nov 02 08:38:13.737312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737241    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istiod-ca-cert\") pod \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") "
Nov 02 08:38:13.737312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737271    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-token\" (UniqueName: \"kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-token\") pod \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") "
Nov 02 08:38:13.737312 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737308    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-kube-api-access-rrf9k\" (UniqueName: \"kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-kube-api-access-rrf9k\") pod \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") "
Nov 02 08:38:13.737659 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737343    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-data\") pod \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") "
Nov 02 08:38:13.737659 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737384    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-certs" pod "7e2ea646-5f4a-4387-8f2c-84fbca7b2089" (UID: "7e2ea646-5f4a-4387-8f2c-84fbca7b2089"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.737659 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737392    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-podinfo\") pod \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\" (UID: \"7e2ea646-5f4a-4387-8f2c-84fbca7b2089\") "
Nov 02 08:38:13.737659 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737553    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-credential-socket" pod "7e2ea646-5f4a-4387-8f2c-84fbca7b2089" (UID: "7e2ea646-5f4a-4387-8f2c-84fbca7b2089"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.737916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737746    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-data" pod "7e2ea646-5f4a-4387-8f2c-84fbca7b2089" (UID: "7e2ea646-5f4a-4387-8f2c-84fbca7b2089"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.737916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737866    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.738052 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737884    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.738052 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.737956    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.738267 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.738235    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-socket" pod "7e2ea646-5f4a-4387-8f2c-84fbca7b2089" (UID: "7e2ea646-5f4a-4387-8f2c-84fbca7b2089"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.738412 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.738344    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istiod-ca-cert" pod "7e2ea646-5f4a-4387-8f2c-84fbca7b2089" (UID: "7e2ea646-5f4a-4387-8f2c-84fbca7b2089"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:38:13.740112 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.740082    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-podinfo" pod "7e2ea646-5f4a-4387-8f2c-84fbca7b2089" (UID: "7e2ea646-5f4a-4387-8f2c-84fbca7b2089"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:38:13.740269 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.740247    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-envoy" pod "7e2ea646-5f4a-4387-8f2c-84fbca7b2089" (UID: "7e2ea646-5f4a-4387-8f2c-84fbca7b2089"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:38:13.740365 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.740348    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-kube-api-access-rrf9k" pod "7e2ea646-5f4a-4387-8f2c-84fbca7b2089" (UID: "7e2ea646-5f4a-4387-8f2c-84fbca7b2089"). InnerVolumeSpecName "kube-api-access-rrf9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:13.740524 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.740506    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-token" pod "7e2ea646-5f4a-4387-8f2c-84fbca7b2089" (UID: "7e2ea646-5f4a-4387-8f2c-84fbca7b2089"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:38:13.839293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.839261    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.839293 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.839289    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.839505 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.839303    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.839505 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.839315    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.839505 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.839328    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrf9k\" (UniqueName: \"kubernetes.io/projected/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-kube-api-access-rrf9k\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:13.839505 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:13.839342    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2ea646-5f4a-4387-8f2c-84fbca7b2089-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:38:14.029065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:14.029030    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"]
Nov 02 08:38:14.040837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:14.040805    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9db802-openshift-default-bd9c7b99f-zzh9k"]
Nov 02 08:38:14.988090 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:38:14.988055    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:38:15.463601 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:15.463566    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2ea646-5f4a-4387-8f2c-84fbca7b2089" path="/var/lib/kubelet/pods/7e2ea646-5f4a-4387-8f2c-84fbca7b2089/volumes"
Nov 02 08:38:15.598462 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:15.598428    4898 scope.go:122] "RemoveContainer" containerID="188eedbf65a49a6be08c0936bf1ddb207af188079a879eaa2e5905dd6a0c89b5"
Nov 02 08:38:15.608408 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:15.608388    4898 scope.go:122] "RemoveContainer" containerID="5d63409a5d50118fa665eba0f3dfbbcbb91b1c4ea68433becad052996d9a8680"
Nov 02 08:38:15.617166 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:15.617145    4898 scope.go:122] "RemoveContainer" containerID="e12e0b5f83ce858c96c9fb1b47c16a6f5618e92947d3215473ddd2fbec166c12"
Nov 02 08:38:15.625395 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:15.625381    4898 scope.go:122] "RemoveContainer" containerID="07a7587e751c01b2e84b4da9e66aa952595929b283c57f1c41b9a6cc99750a7b"
Nov 02 08:38:15.633492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:15.633474    4898 scope.go:122] "RemoveContainer" containerID="c3b92c1956a996d8c15dc222ed09569473c2968ba6883515370755da4c04cfa2"
Nov 02 08:38:15.641536 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:15.641516    4898 scope.go:122] "RemoveContainer" containerID="c10d8ea9f9fe7ce6f681878658626b2c5b5359bc96411f65476142fd026785a1"
Nov 02 08:38:15.649839 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:15.649822    4898 scope.go:122] "RemoveContainer" containerID="a86515ab80e66cdbb046040ac9825b43db427c3bf01af511b875786ed4af562b"
Nov 02 08:38:15.657884 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:15.657867    4898 scope.go:122] "RemoveContainer" containerID="c808b2f49791e9cfdbb343998d7d20a7e077486569d4a11bef50c79baecb7cc6"
Nov 02 08:38:26.219114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:26.219037    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5698f55746-5klb6_37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf/authorino/0.log"
Nov 02 08:38:26.473303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:26.473227    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-547hn_eddc764b-21f2-4cd2-b5b3-d331ad9c131d/manager/0.log"
Nov 02 08:38:26.678995 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:26.678964    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-nh2gw_b78e88ff-b93b-48c6-b466-cac6e05256d9/manager/0.log"
Nov 02 08:38:27.223956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:27.223915    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-br6sk_d2c780cc-30a6-43ed-afe5-77b774bb026f/manager/0.log"
Nov 02 08:38:27.711162 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:27.711126    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pvwn5_a2a4d97a-4060-429a-81e6-dffbfd834a6d/manager/0.log"
Nov 02 08:38:30.132803 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:38:30.132768    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-rcnpx_7c663e0a-d1b2-4ea4-91d3-fe88764f387d/manager/0.log"
Nov 02 08:38:44.988652 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:38:44.988613    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:39:10.898180 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:10.898145    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:39:10.907671 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:10.907634    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:39:14.991580 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:39:14.991532    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:39:36.033966 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:36.033926    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"]
Nov 02 08:39:36.043848 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:36.043821    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1258bc51-sthqm"]
Nov 02 08:39:37.464568 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:37.464525    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9105f772-fef3-420e-990b-f9a0658ca143" path="/var/lib/kubelet/pods/9105f772-fef3-420e-990b-f9a0658ca143/volumes"
Nov 02 08:39:42.032153 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:42.032114    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"]
Nov 02 08:39:42.043760 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:42.043717    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1b0c29-fg75p"]
Nov 02 08:39:43.464323 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:43.464287    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f87f67d-575a-45df-8e5f-205e1eca5a60" path="/var/lib/kubelet/pods/1f87f67d-575a-45df-8e5f-205e1eca5a60/volumes"
Nov 02 08:39:44.988784 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:39:44.988742    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:39:47.031589 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:47.031552    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"]
Nov 02 08:39:47.043513 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:47.043482    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8b63b1-jj5md"]
Nov 02 08:39:47.471618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:47.471575    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9a22a6-ab15-473c-b6cc-0a00cc8ad371" path="/var/lib/kubelet/pods/fd9a22a6-ab15-473c-b6cc-0a00cc8ad371/volumes"
Nov 02 08:39:52.047856 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:52.047760    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"]
Nov 02 08:39:52.059702 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:52.059668    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80122865-k95qx"]
Nov 02 08:39:53.463015 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:39:53.462976    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7254db6-7bd2-4cea-8fa5-e2f7354fc9af" path="/var/lib/kubelet/pods/e7254db6-7bd2-4cea-8fa5-e2f7354fc9af/volumes"
Nov 02 08:40:03.064463 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:40:03.064422    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"]
Nov 02 08:40:03.075283 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:40:03.075248    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46aca450-m4gp7"]
Nov 02 08:40:03.463421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:40:03.463388    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0127b61-caef-4959-b4ad-f34e72e25bde" path="/var/lib/kubelet/pods/f0127b61-caef-4959-b4ad-f34e72e25bde/volumes"
Nov 02 08:40:14.032615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:40:14.032574    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"]
Nov 02 08:40:14.045553 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:40:14.045519    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9e33ca-sdhzs"]
Nov 02 08:40:14.988757 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:40:14.988708    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:40:15.464362 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:40:15.464323    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e98831c-cc04-405f-a7fb-d8c19bf60bcb" path="/var/lib/kubelet/pods/0e98831c-cc04-405f-a7fb-d8c19bf60bcb/volumes"
Nov 02 08:40:38.126914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:40:38.126044    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:40:38.126914 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:40:38.126152    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:40:44.994488 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:40:44.994440    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:41:10.030034 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:10.029997    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"]
Nov 02 08:41:10.042205 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:10.042172    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1807b8-4545h"]
Nov 02 08:41:11.462984 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:11.462953    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938815d3-2163-4f72-9a95-58c1b084bbf2" path="/var/lib/kubelet/pods/938815d3-2163-4f72-9a95-58c1b084bbf2/volumes"
Nov 02 08:41:14.988719 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:41:14.988682    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:41:15.776366 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:15.776333    4898 scope.go:122] "RemoveContainer" containerID="6038d72325187addc901b5e3177fb9dffc4d2010a5c26456d88091bf561ebc34"
Nov 02 08:41:15.786022 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:15.785993    4898 scope.go:122] "RemoveContainer" containerID="8d14175a9523a60a2a09467876b4b2e71a5393242151bcf22d3561775dc6ef7c"
Nov 02 08:41:15.794275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:15.794250    4898 scope.go:122] "RemoveContainer" containerID="79520f955e496390946406a90a507e901c91d57dc40dd0fc89097de0b0b2eec0"
Nov 02 08:41:15.802107 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:15.802087    4898 scope.go:122] "RemoveContainer" containerID="7567306e763f0bb941426d21b31a055fe856d715a3f44c1015cfa79413ae46bb"
Nov 02 08:41:15.810336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:15.810320    4898 scope.go:122] "RemoveContainer" containerID="cb079292079498cca01cd4ceb6d11479446622eaac2af0145a10846b9d58a4e4"
Nov 02 08:41:15.818088 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:15.818068    4898 scope.go:122] "RemoveContainer" containerID="cee1ce63261ce5f00dfd5c2dc033bf14beecda3deb7d760908489df5cc1b54c8"
Nov 02 08:41:21.041742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:21.041708    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"]
Nov 02 08:41:21.052604 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:21.052576    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641991db-k6f96"]
Nov 02 08:41:21.463974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:21.463937    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="567fbe26-d5e3-498e-a102-1f7ff9e9f031" path="/var/lib/kubelet/pods/567fbe26-d5e3-498e-a102-1f7ff9e9f031/volumes"
Nov 02 08:41:44.987660 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:41:44.987618    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:41:56.038431 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:56.038386    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"]
Nov 02 08:41:56.050291 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:56.050250    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e86193d-5nmpz"]
Nov 02 08:41:57.463340 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:41:57.463308    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47b8fed1-75bc-4c61-a734-0d48d231db82" path="/var/lib/kubelet/pods/47b8fed1-75bc-4c61-a734-0d48d231db82/volumes"
Nov 02 08:42:09.035380 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:09.029550    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"]
Nov 02 08:42:09.045586 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:09.045552    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd7d3351-b2m8h"]
Nov 02 08:42:09.463352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:09.463308    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34165fb1-dda1-403c-9fc7-0251e8769a5e" path="/var/lib/kubelet/pods/34165fb1-dda1-403c-9fc7-0251e8769a5e/volumes"
Nov 02 08:42:14.987749 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:42:14.987708    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:42:15.859640 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:15.859606    4898 scope.go:122] "RemoveContainer" containerID="40af73718f78ee132e00c35ab9768526637ffaa6930fe09df7d325a7f2584df4"
Nov 02 08:42:26.031603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:26.031556    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"]
Nov 02 08:42:26.044430 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:26.044394    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81019a41-qk6qk"]
Nov 02 08:42:27.542974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:27.541504    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="308d8aaf-05e2-4c02-9a17-2b5442e9a188" path="/var/lib/kubelet/pods/308d8aaf-05e2-4c02-9a17-2b5442e9a188/volumes"
Nov 02 08:42:36.771665 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.771603    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"]
Nov 02 08:42:36.773030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.772996    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7e2ea646-5f4a-4387-8f2c-84fbca7b2089" containerName="istio-proxy"
Nov 02 08:42:36.773030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.773029    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2ea646-5f4a-4387-8f2c-84fbca7b2089" containerName="istio-proxy"
Nov 02 08:42:36.773235 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.773204    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2ea646-5f4a-4387-8f2c-84fbca7b2089" containerName="istio-proxy"
Nov 02 08:42:36.776639 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.776614    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:36.796683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.796660    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-1b623b-openshift-default-dockercfg-nzzmg\""
Nov 02 08:42:36.811771 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.811742    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"]
Nov 02 08:42:36.946512 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.946469    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-credential-socket\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:36.947036 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.947000    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-envoy\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:36.947369 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.947345    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqrbr\" (UniqueName: \"kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-kube-api-access-jqrbr\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:36.948485 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.948462    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istiod-ca-cert\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:36.948726 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.948709    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-certs\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:36.948903 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.948867    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-token\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:36.949064 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.949048    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-podinfo\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:36.949196 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.949181    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-data\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:36.949351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:36.949331    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-socket\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.031878 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.031778    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"]
Nov 02 08:42:37.035484 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.035446    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6e80bf-djb7x"]
Nov 02 08:42:37.050600 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.050570    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-certs\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.050732 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.050610    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-token\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.050783 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.050747    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-podinfo\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.050834 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.050804    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-data\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.050884 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.050864    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-socket\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.050986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.050914    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-credential-socket\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.050986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.050950    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-envoy\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.051375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.050975    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrbr\" (UniqueName: \"kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-kube-api-access-jqrbr\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.051375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.051294    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-data\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.051375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.051054    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-certs\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.051375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.051338    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istiod-ca-cert\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.051375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.051258    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-socket\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.051718 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.051413    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-credential-socket\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.051930 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.051908    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istiod-ca-cert\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.053246 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.053220    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-envoy\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.053425 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.053407    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-podinfo\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.074073 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.074046    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqrbr\" (UniqueName: \"kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-kube-api-access-jqrbr\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.074202 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.074085    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-token\") pod \"e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") " pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.091625 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.091592    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:37.407675 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.407646    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"]
Nov 02 08:42:37.409446 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:42:37.409416    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f70659_a519_4f21_b29d_4e8ba2b4cf66.slice/crio-00e00d52a961ec810f5e93b1206dcee333618751351e82127833bc81f5f7bc3d WatchSource:0}: Error finding container 00e00d52a961ec810f5e93b1206dcee333618751351e82127833bc81f5f7bc3d: Status 404 returned error can't find the container with id 00e00d52a961ec810f5e93b1206dcee333618751351e82127833bc81f5f7bc3d
Nov 02 08:42:37.411829 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.411795    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:42:37.411955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.411864    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:42:37.411955 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.411915    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:42:37.463881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.463849    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c11b851c-efbb-4f8b-8dab-d37b56776c86" path="/var/lib/kubelet/pods/c11b851c-efbb-4f8b-8dab-d37b56776c86/volumes"
Nov 02 08:42:37.534363 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.534322    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" event={"ID":"62f70659-a519-4f21-b29d-4e8ba2b4cf66","Type":"ContainerStarted","Data":"a5aa1181d9410ceee46ac93620b83b05f94da18353ae93b457d758ac7ba023f3"}
Nov 02 08:42:37.534582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.534387    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" event={"ID":"62f70659-a519-4f21-b29d-4e8ba2b4cf66","Type":"ContainerStarted","Data":"00e00d52a961ec810f5e93b1206dcee333618751351e82127833bc81f5f7bc3d"}
Nov 02 08:42:37.579903 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:37.579828    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" podStartSLOduration=1.579810328 podStartE2EDuration="1.579810328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:42:37.578475379 +0000 UTC m=+4708.268802367" watchObservedRunningTime="2026-11-02 08:42:37.579810328 +0000 UTC m=+4708.270137292"
Nov 02 08:42:38.092344 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:38.092305    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:38.097453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:38.097423    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:38.542383 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:38.542345    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:38.543179 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:38.543155    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:42.087369 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.087324    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"]
Nov 02 08:42:42.091547 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.091521    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"
Nov 02 08:42:42.111691 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.111664    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"]
Nov 02 08:42:42.120837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.120799    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-1b623b\""
Nov 02 08:42:42.120837 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.120818    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-43c72653-dockercfg-jtg7t\""
Nov 02 08:42:42.233937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.232614    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3fb475-4d09-4a86-80ab-10e6d38a9163-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv\" (UID: \"1f3fb475-4d09-4a86-80ab-10e6d38a9163\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"
Nov 02 08:42:42.233937 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.232704    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppz8h\" (UniqueName: \"kubernetes.io/projected/1f3fb475-4d09-4a86-80ab-10e6d38a9163-kube-api-access-ppz8h\") pod \"payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv\" (UID: \"1f3fb475-4d09-4a86-80ab-10e6d38a9163\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"
Nov 02 08:42:42.236324 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.235366    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"]
Nov 02 08:42:42.240911 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.239791    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"
Nov 02 08:42:42.242159 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.241745    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"]
Nov 02 08:42:42.333778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.333737    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb904c60-e123-45df-b8ef-b44a83dc12a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8\" (UID: \"eb904c60-e123-45df-b8ef-b44a83dc12a7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"
Nov 02 08:42:42.334025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.333787    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m59jc\" (UniqueName: \"kubernetes.io/projected/eb904c60-e123-45df-b8ef-b44a83dc12a7-kube-api-access-m59jc\") pod \"payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8\" (UID: \"eb904c60-e123-45df-b8ef-b44a83dc12a7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"
Nov 02 08:42:42.334025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.333876    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3fb475-4d09-4a86-80ab-10e6d38a9163-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv\" (UID: \"1f3fb475-4d09-4a86-80ab-10e6d38a9163\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"
Nov 02 08:42:42.334025 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.333971    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppz8h\" (UniqueName: \"kubernetes.io/projected/1f3fb475-4d09-4a86-80ab-10e6d38a9163-kube-api-access-ppz8h\") pod \"payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv\" (UID: \"1f3fb475-4d09-4a86-80ab-10e6d38a9163\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"
Nov 02 08:42:42.334560 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.334542    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3fb475-4d09-4a86-80ab-10e6d38a9163-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv\" (UID: \"1f3fb475-4d09-4a86-80ab-10e6d38a9163\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"
Nov 02 08:42:42.357714 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.357644    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppz8h\" (UniqueName: \"kubernetes.io/projected/1f3fb475-4d09-4a86-80ab-10e6d38a9163-kube-api-access-ppz8h\") pod \"payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv\" (UID: \"1f3fb475-4d09-4a86-80ab-10e6d38a9163\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"
Nov 02 08:42:42.405642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.405614    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"
Nov 02 08:42:42.434957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.434931    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb904c60-e123-45df-b8ef-b44a83dc12a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8\" (UID: \"eb904c60-e123-45df-b8ef-b44a83dc12a7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"
Nov 02 08:42:42.435092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.434973    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m59jc\" (UniqueName: \"kubernetes.io/projected/eb904c60-e123-45df-b8ef-b44a83dc12a7-kube-api-access-m59jc\") pod \"payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8\" (UID: \"eb904c60-e123-45df-b8ef-b44a83dc12a7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"
Nov 02 08:42:42.435629 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.435606    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb904c60-e123-45df-b8ef-b44a83dc12a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8\" (UID: \"eb904c60-e123-45df-b8ef-b44a83dc12a7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"
Nov 02 08:42:42.460445 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.460060    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m59jc\" (UniqueName: \"kubernetes.io/projected/eb904c60-e123-45df-b8ef-b44a83dc12a7-kube-api-access-m59jc\") pod \"payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8\" (UID: \"eb904c60-e123-45df-b8ef-b44a83dc12a7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"
Nov 02 08:42:42.560991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.560960    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"
Nov 02 08:42:42.734593 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.734559    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"]
Nov 02 08:42:42.735624 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:42:42.735596    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3fb475_4d09_4a86_80ab_10e6d38a9163.slice/crio-e1b2cf80f13479d3a68404749922fc7c617afe1958ebce7acc1c3c2944fb9db0 WatchSource:0}: Error finding container e1b2cf80f13479d3a68404749922fc7c617afe1958ebce7acc1c3c2944fb9db0: Status 404 returned error can't find the container with id e1b2cf80f13479d3a68404749922fc7c617afe1958ebce7acc1c3c2944fb9db0
Nov 02 08:42:42.886532 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:42.886505    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"]
Nov 02 08:42:42.888351 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:42:42.888328    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb904c60_e123_45df_b8ef_b44a83dc12a7.slice/crio-4facdf1ccebb17daaeb7dcd61a6593142e5e44ca72de7be9ac553f1ff1f2ffd4 WatchSource:0}: Error finding container 4facdf1ccebb17daaeb7dcd61a6593142e5e44ca72de7be9ac553f1ff1f2ffd4: Status 404 returned error can't find the container with id 4facdf1ccebb17daaeb7dcd61a6593142e5e44ca72de7be9ac553f1ff1f2ffd4
Nov 02 08:42:43.580582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:43.580536    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8" event={"ID":"eb904c60-e123-45df-b8ef-b44a83dc12a7","Type":"ContainerStarted","Data":"c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30"}
Nov 02 08:42:43.581082 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:43.580641    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"
Nov 02 08:42:43.581082 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:43.580665    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8" event={"ID":"eb904c60-e123-45df-b8ef-b44a83dc12a7","Type":"ContainerStarted","Data":"4facdf1ccebb17daaeb7dcd61a6593142e5e44ca72de7be9ac553f1ff1f2ffd4"}
Nov 02 08:42:43.582563 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:43.582530    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv" event={"ID":"1f3fb475-4d09-4a86-80ab-10e6d38a9163","Type":"ContainerStarted","Data":"cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df"}
Nov 02 08:42:43.582721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:43.582574    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv" event={"ID":"1f3fb475-4d09-4a86-80ab-10e6d38a9163","Type":"ContainerStarted","Data":"e1b2cf80f13479d3a68404749922fc7c617afe1958ebce7acc1c3c2944fb9db0"}
Nov 02 08:42:43.582721 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:43.582616    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"
Nov 02 08:42:43.622459 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:43.622402    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8" podStartSLOduration=1.622385564 podStartE2EDuration="1.622385564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:42:43.620396411 +0000 UTC m=+4714.310723376" watchObservedRunningTime="2026-11-02 08:42:43.622385564 +0000 UTC m=+4714.312712530"
Nov 02 08:42:43.661796 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:43.661738    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv" podStartSLOduration=1.661725329 podStartE2EDuration="1.661725329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:42:43.660511366 +0000 UTC m=+4714.350838334" watchObservedRunningTime="2026-11-02 08:42:43.661725329 +0000 UTC m=+4714.352052293"
Nov 02 08:42:44.988189 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:42:44.988151    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:42:45.028986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:45.028949    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"]
Nov 02 08:42:45.041122 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:45.041093    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c7f302-fl54g"]
Nov 02 08:42:45.467279 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:45.467235    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459a5dd3-a78c-4ba7-9338-e36624d2584c" path="/var/lib/kubelet/pods/459a5dd3-a78c-4ba7-9338-e36624d2584c/volumes"
Nov 02 08:42:54.590046 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:54.590005    4898 patch_prober.go:29] interesting pod/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:42:54.590618 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:54.590094    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:42:54.593421 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:54.593396    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"
Nov 02 08:42:54.593547 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:54.593478    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"
Nov 02 08:42:55.665724 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:55.665691    4898 generic.go:356] "Generic (PLEG): container finished" podID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerID="a5aa1181d9410ceee46ac93620b83b05f94da18353ae93b457d758ac7ba023f3" exitCode=0
Nov 02 08:42:55.666212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:55.665765    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" event={"ID":"62f70659-a519-4f21-b29d-4e8ba2b4cf66","Type":"ContainerDied","Data":"a5aa1181d9410ceee46ac93620b83b05f94da18353ae93b457d758ac7ba023f3"}
Nov 02 08:42:55.666212 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:55.666191    4898 scope.go:122] "RemoveContainer" containerID="a5aa1181d9410ceee46ac93620b83b05f94da18353ae93b457d758ac7ba023f3"
Nov 02 08:42:55.667482 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:55.667451    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:42:55.667545 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:55.667523    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:42:55.667584 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:55.667564    4898 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 08:42:56.686387 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:56.686345    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" event={"ID":"62f70659-a519-4f21-b29d-4e8ba2b4cf66","Type":"ContainerStarted","Data":"9087ad5cd0e66d0b2bd2b72b9eec579a37c88923776d067a6fb7e86d51bc5620"}
Nov 02 08:42:57.092674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:57.092587    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:42:57.092840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:57.092813    4898 patch_prober.go:29] interesting pod/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body=
Nov 02 08:42:57.092916 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:57.092859    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused"
Nov 02 08:42:58.092523 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:58.092488    4898 patch_prober.go:29] interesting pod/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body=
Nov 02 08:42:58.093030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:58.092557    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused"
Nov 02 08:42:59.092582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:59.092544    4898 patch_prober.go:29] interesting pod/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body=
Nov 02 08:42:59.093015 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:42:59.092609    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused"
Nov 02 08:43:00.092544 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:00.092504    4898 patch_prober.go:29] interesting pod/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body=
Nov 02 08:43:00.092743 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:00.092562    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused"
Nov 02 08:43:01.092416 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:01.092378    4898 patch_prober.go:29] interesting pod/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused" start-of-body=
Nov 02 08:43:01.092626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:01.092437    4898 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.167:15021/healthz/ready\": dial tcp 10.132.1.167:15021: connect: connection refused"
Nov 02 08:43:02.095630 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:02.095597    4898 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:43:02.096073 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:02.095932    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:43:02.096603 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:02.096583    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:43:14.988017 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:43:14.987979    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:43:15.882773 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:15.882738    4898 scope.go:122] "RemoveContainer" containerID="9d06918414fd2730c1cf0619abfba872f7976dfea7fdc72dc5e85ed753845339"
Nov 02 08:43:15.892000 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:15.891970    4898 scope.go:122] "RemoveContainer" containerID="d50b7c962381a0ef63c107723cbc266b51948720f5960488ef2b6ce5e46d2f88"
Nov 02 08:43:15.900284 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:15.900262    4898 scope.go:122] "RemoveContainer" containerID="80d636c23a604c8ac922799b0cf030f2b25da8e8ac3f65b7f056370795b0114a"
Nov 02 08:43:39.584582 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.584550    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"]
Nov 02 08:43:39.588206 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.588185    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:39.616624 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.616445    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:43:39.616624 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.616475    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 08:43:39.616624 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.616551    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:43:39.616624 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.616595    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:43:39.620056 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.620030    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"]
Nov 02 08:43:39.702012 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.701975    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8966ac03-sjnxt\" (UID: \"85827207-d34c-472d-af30-2bc2182f52ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:39.702187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.702085    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl4lt\" (UniqueName: \"kubernetes.io/projected/85827207-d34c-472d-af30-2bc2182f52ff-kube-api-access-gl4lt\") pod \"maas-api-revoke-sub-8966ac03-sjnxt\" (UID: \"85827207-d34c-472d-af30-2bc2182f52ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:39.702187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.702185    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8966ac03-sjnxt\" (UID: \"85827207-d34c-472d-af30-2bc2182f52ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:39.803684 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.803643    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8966ac03-sjnxt\" (UID: \"85827207-d34c-472d-af30-2bc2182f52ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:39.803938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.803730    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8966ac03-sjnxt\" (UID: \"85827207-d34c-472d-af30-2bc2182f52ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:39.803938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.803793    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4lt\" (UniqueName: \"kubernetes.io/projected/85827207-d34c-472d-af30-2bc2182f52ff-kube-api-access-gl4lt\") pod \"maas-api-revoke-sub-8966ac03-sjnxt\" (UID: \"85827207-d34c-472d-af30-2bc2182f52ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:39.804473 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.804450    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8966ac03-sjnxt\" (UID: \"85827207-d34c-472d-af30-2bc2182f52ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:39.806247 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.806218    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8966ac03-sjnxt\" (UID: \"85827207-d34c-472d-af30-2bc2182f52ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:39.827956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.827925    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl4lt\" (UniqueName: \"kubernetes.io/projected/85827207-d34c-472d-af30-2bc2182f52ff-kube-api-access-gl4lt\") pod \"maas-api-revoke-sub-8966ac03-sjnxt\" (UID: \"85827207-d34c-472d-af30-2bc2182f52ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:39.900910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:39.900864    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:40.201238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:40.201205    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"]
Nov 02 08:43:40.202057 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:43:40.202031    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85827207_d34c_472d_af30_2bc2182f52ff.slice/crio-3977e409b30e755af1972e0ac732fc610a9e631280baabe44d289ed47a44d561 WatchSource:0}: Error finding container 3977e409b30e755af1972e0ac732fc610a9e631280baabe44d289ed47a44d561: Status 404 returned error can't find the container with id 3977e409b30e755af1972e0ac732fc610a9e631280baabe44d289ed47a44d561
Nov 02 08:43:40.984621 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:40.984582    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt" event={"ID":"85827207-d34c-472d-af30-2bc2182f52ff","Type":"ContainerStarted","Data":"cbb3c67376b4fceaf8632784e8f1f5e37a014753099942efdffbcbd443af891d"}
Nov 02 08:43:40.985144 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:40.984638    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt" event={"ID":"85827207-d34c-472d-af30-2bc2182f52ff","Type":"ContainerStarted","Data":"3977e409b30e755af1972e0ac732fc610a9e631280baabe44d289ed47a44d561"}
Nov 02 08:43:41.026327 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:41.026262    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt" podStartSLOduration=2.026248259 podStartE2EDuration="2.026248259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:43:41.024869701 +0000 UTC m=+4771.715196666" watchObservedRunningTime="2026-11-02 08:43:41.026248259 +0000 UTC m=+4771.716575288"
Nov 02 08:43:41.991613 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:41.991577    4898 generic.go:356] "Generic (PLEG): container finished" podID="85827207-d34c-472d-af30-2bc2182f52ff" containerID="cbb3c67376b4fceaf8632784e8f1f5e37a014753099942efdffbcbd443af891d" exitCode=0
Nov 02 08:43:41.992048 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:41.991627    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt" event={"ID":"85827207-d34c-472d-af30-2bc2182f52ff","Type":"ContainerDied","Data":"cbb3c67376b4fceaf8632784e8f1f5e37a014753099942efdffbcbd443af891d"}
Nov 02 08:43:43.153301 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:43.153270    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:43.238443 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:43.238407    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-xks-ca\") pod \"85827207-d34c-472d-af30-2bc2182f52ff\" (UID: \"85827207-d34c-472d-af30-2bc2182f52ff\") "
Nov 02 08:43:43.238615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:43.238544    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85827207-d34c-472d-af30-2bc2182f52ff-kube-api-access-gl4lt\" (UniqueName: \"kubernetes.io/projected/85827207-d34c-472d-af30-2bc2182f52ff-kube-api-access-gl4lt\") pod \"85827207-d34c-472d-af30-2bc2182f52ff\" (UID: \"85827207-d34c-472d-af30-2bc2182f52ff\") "
Nov 02 08:43:43.238615 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:43.238583    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-openshift-ca\") pod \"85827207-d34c-472d-af30-2bc2182f52ff\" (UID: \"85827207-d34c-472d-af30-2bc2182f52ff\") "
Nov 02 08:43:43.239005 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:43.238978    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-openshift-ca" pod "85827207-d34c-472d-af30-2bc2182f52ff" (UID: "85827207-d34c-472d-af30-2bc2182f52ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:43:43.240778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:43.240745    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85827207-d34c-472d-af30-2bc2182f52ff-kube-api-access-gl4lt" pod "85827207-d34c-472d-af30-2bc2182f52ff" (UID: "85827207-d34c-472d-af30-2bc2182f52ff"). InnerVolumeSpecName "kube-api-access-gl4lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:43:43.240778 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:43.240755    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-xks-ca" pod "85827207-d34c-472d-af30-2bc2182f52ff" (UID: "85827207-d34c-472d-af30-2bc2182f52ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:43:43.339566 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:43.339488    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:43:43.339566 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:43.339521    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl4lt\" (UniqueName: \"kubernetes.io/projected/85827207-d34c-472d-af30-2bc2182f52ff-kube-api-access-gl4lt\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:43:43.339566 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:43.339533    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85827207-d34c-472d-af30-2bc2182f52ff-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:43:44.006626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:44.006586    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt" event={"ID":"85827207-d34c-472d-af30-2bc2182f52ff","Type":"ContainerDied","Data":"3977e409b30e755af1972e0ac732fc610a9e631280baabe44d289ed47a44d561"}
Nov 02 08:43:44.006626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:44.006623    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"
Nov 02 08:43:44.006852 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:44.006641    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3977e409b30e755af1972e0ac732fc610a9e631280baabe44d289ed47a44d561"
Nov 02 08:43:44.988319 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:43:44.988283    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:43:56.787349 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:56.787307    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"]
Nov 02 08:43:56.824101 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:43:56.824066    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"]
Nov 02 08:44:01.788580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:01.788525    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8" podUID="eb904c60-e123-45df-b8ef-b44a83dc12a7" containerName="payload-processing" containerID="cri-o://c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30" gracePeriod=25
Nov 02 08:44:01.824957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:01.824884    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv" podUID="1f3fb475-4d09-4a86-80ab-10e6d38a9163" containerName="payload-pre-processing" containerID="cri-o://cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df" gracePeriod=25
Nov 02 08:44:04.592529 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:04.592487    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8" podUID="eb904c60-e123-45df-b8ef-b44a83dc12a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:44:04.592966 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:04.592487    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv" podUID="1f3fb475-4d09-4a86-80ab-10e6d38a9163" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:44:10.941131 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:10.941096    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:44:10.950554 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:10.950532    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:44:14.592055 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:14.592010    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8" podUID="eb904c60-e123-45df-b8ef-b44a83dc12a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:44:14.592494 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:14.592012    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv" podUID="1f3fb475-4d09-4a86-80ab-10e6d38a9163" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 08:44:14.989345 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:44:14.989307    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:44:15.931661 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:15.931627    4898 scope.go:122] "RemoveContainer" containerID="2796bfb6e348788ecc3b4268bfdfc094782303027d4470b3bd798ad41890aeab"
Nov 02 08:44:15.940803 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:15.940782    4898 scope.go:122] "RemoveContainer" containerID="e535f7425024c8b82fd883d7c3f7a85547a7cd0fe5ef8d420b20cee397c2f8b4"
Nov 02 08:44:15.948985 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:15.948965    4898 scope.go:122] "RemoveContainer" containerID="0fbda3ada91666107a119c96feb34f4bd6b5ce5a6994f801136357b3d5f9b68c"
Nov 02 08:44:22.087913 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.087864    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"
Nov 02 08:44:22.107629 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.107608    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"
Nov 02 08:44:22.161155 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.161126    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb904c60-e123-45df-b8ef-b44a83dc12a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb904c60-e123-45df-b8ef-b44a83dc12a7-plugins-config-volume\") pod \"eb904c60-e123-45df-b8ef-b44a83dc12a7\" (UID: \"eb904c60-e123-45df-b8ef-b44a83dc12a7\") "
Nov 02 08:44:22.161292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.161225    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb904c60-e123-45df-b8ef-b44a83dc12a7-kube-api-access-m59jc\" (UniqueName: \"kubernetes.io/projected/eb904c60-e123-45df-b8ef-b44a83dc12a7-kube-api-access-m59jc\") pod \"eb904c60-e123-45df-b8ef-b44a83dc12a7\" (UID: \"eb904c60-e123-45df-b8ef-b44a83dc12a7\") "
Nov 02 08:44:22.161498 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.161473    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb904c60-e123-45df-b8ef-b44a83dc12a7-plugins-config-volume" pod "eb904c60-e123-45df-b8ef-b44a83dc12a7" (UID: "eb904c60-e123-45df-b8ef-b44a83dc12a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:44:22.163256 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.163235    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb904c60-e123-45df-b8ef-b44a83dc12a7-kube-api-access-m59jc" pod "eb904c60-e123-45df-b8ef-b44a83dc12a7" (UID: "eb904c60-e123-45df-b8ef-b44a83dc12a7"). InnerVolumeSpecName "kube-api-access-m59jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:44:22.240279 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.240245    4898 generic.go:356] "Generic (PLEG): container finished" podID="1f3fb475-4d09-4a86-80ab-10e6d38a9163" containerID="cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df" exitCode=0
Nov 02 08:44:22.240466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.240319    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"
Nov 02 08:44:22.240466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.240331    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv" event={"ID":"1f3fb475-4d09-4a86-80ab-10e6d38a9163","Type":"ContainerDied","Data":"cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df"}
Nov 02 08:44:22.240466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.240395    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv" event={"ID":"1f3fb475-4d09-4a86-80ab-10e6d38a9163","Type":"ContainerDied","Data":"e1b2cf80f13479d3a68404749922fc7c617afe1958ebce7acc1c3c2944fb9db0"}
Nov 02 08:44:22.240466 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.240426    4898 scope.go:122] "RemoveContainer" containerID="cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df"
Nov 02 08:44:22.241958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.241934    4898 generic.go:356] "Generic (PLEG): container finished" podID="eb904c60-e123-45df-b8ef-b44a83dc12a7" containerID="c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30" exitCode=0
Nov 02 08:44:22.242084 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.241985    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"
Nov 02 08:44:22.242135 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.241990    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8" event={"ID":"eb904c60-e123-45df-b8ef-b44a83dc12a7","Type":"ContainerDied","Data":"c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30"}
Nov 02 08:44:22.242135 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.242124    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8" event={"ID":"eb904c60-e123-45df-b8ef-b44a83dc12a7","Type":"ContainerDied","Data":"4facdf1ccebb17daaeb7dcd61a6593142e5e44ca72de7be9ac553f1ff1f2ffd4"}
Nov 02 08:44:22.251597 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.251497    4898 scope.go:122] "RemoveContainer" containerID="cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df"
Nov 02 08:44:22.251858 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:44:22.251827    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df\": container with ID starting with cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df not found: ID does not exist" containerID="cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df"
Nov 02 08:44:22.252002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.251864    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df"} err="failed to get container status \"cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df\": rpc error: code = NotFound desc = could not find container \"cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df\": container with ID starting with cb94e21e6b1705a6ee11dc9e94fbf5dc7e5fbff7f2bb286a98b1e426cb56c0df not found: ID does not exist"
Nov 02 08:44:22.252002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.251884    4898 scope.go:122] "RemoveContainer" containerID="c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30"
Nov 02 08:44:22.259997 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.259975    4898 scope.go:122] "RemoveContainer" containerID="c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30"
Nov 02 08:44:22.260256 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:44:22.260230    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30\": container with ID starting with c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30 not found: ID does not exist" containerID="c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30"
Nov 02 08:44:22.260349 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.260259    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30"} err="failed to get container status \"c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30\": rpc error: code = NotFound desc = could not find container \"c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30\": container with ID starting with c3932429a999b6f6ecf338cb2b66aa0217969a13791ed5de2957b618c465cd30 not found: ID does not exist"
Nov 02 08:44:22.262626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.262609    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3fb475-4d09-4a86-80ab-10e6d38a9163-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3fb475-4d09-4a86-80ab-10e6d38a9163-plugins-config-volume\") pod \"1f3fb475-4d09-4a86-80ab-10e6d38a9163\" (UID: \"1f3fb475-4d09-4a86-80ab-10e6d38a9163\") "
Nov 02 08:44:22.262697 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.262658    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3fb475-4d09-4a86-80ab-10e6d38a9163-kube-api-access-ppz8h\" (UniqueName: \"kubernetes.io/projected/1f3fb475-4d09-4a86-80ab-10e6d38a9163-kube-api-access-ppz8h\") pod \"1f3fb475-4d09-4a86-80ab-10e6d38a9163\" (UID: \"1f3fb475-4d09-4a86-80ab-10e6d38a9163\") "
Nov 02 08:44:22.262950 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.262932    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3fb475-4d09-4a86-80ab-10e6d38a9163-plugins-config-volume" pod "1f3fb475-4d09-4a86-80ab-10e6d38a9163" (UID: "1f3fb475-4d09-4a86-80ab-10e6d38a9163"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:44:22.263006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.262988    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb904c60-e123-45df-b8ef-b44a83dc12a7-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:44:22.263044 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.263013    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m59jc\" (UniqueName: \"kubernetes.io/projected/eb904c60-e123-45df-b8ef-b44a83dc12a7-kube-api-access-m59jc\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:44:22.264793 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.264774    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3fb475-4d09-4a86-80ab-10e6d38a9163-kube-api-access-ppz8h" pod "1f3fb475-4d09-4a86-80ab-10e6d38a9163" (UID: "1f3fb475-4d09-4a86-80ab-10e6d38a9163"). InnerVolumeSpecName "kube-api-access-ppz8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:44:22.292910 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.292875    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"]
Nov 02 08:44:22.295463 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.295441    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1b623b-c5bc974b7984m8"]
Nov 02 08:44:22.363877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.363820    4898 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3fb475-4d09-4a86-80ab-10e6d38a9163-plugins-config-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:44:22.363877 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.363842    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppz8h\" (UniqueName: \"kubernetes.io/projected/1f3fb475-4d09-4a86-80ab-10e6d38a9163-kube-api-access-ppz8h\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:44:22.585988 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.585945    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"]
Nov 02 08:44:22.590327 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:22.590304    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1b623b-85fb8997pv"]
Nov 02 08:44:23.463709 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:23.463673    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3fb475-4d09-4a86-80ab-10e6d38a9163" path="/var/lib/kubelet/pods/1f3fb475-4d09-4a86-80ab-10e6d38a9163/volumes"
Nov 02 08:44:23.464150 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:23.464133    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb904c60-e123-45df-b8ef-b44a83dc12a7" path="/var/lib/kubelet/pods/eb904c60-e123-45df-b8ef-b44a83dc12a7/volumes"
Nov 02 08:44:44.988690 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:44:44.988649    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:44:55.295408 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:55.295372    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"]
Nov 02 08:44:55.295939 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:44:55.295598    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy" containerID="cri-o://9087ad5cd0e66d0b2bd2b72b9eec579a37c88923776d067a6fb7e86d51bc5620" gracePeriod=30
Nov 02 08:45:00.521110 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.521066    4898 generic.go:356] "Generic (PLEG): container finished" podID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerID="9087ad5cd0e66d0b2bd2b72b9eec579a37c88923776d067a6fb7e86d51bc5620" exitCode=0
Nov 02 08:45:00.521549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.521139    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" event={"ID":"62f70659-a519-4f21-b29d-4e8ba2b4cf66","Type":"ContainerDied","Data":"9087ad5cd0e66d0b2bd2b72b9eec579a37c88923776d067a6fb7e86d51bc5620"}
Nov 02 08:45:00.521549 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.521256    4898 scope.go:122] "RemoveContainer" containerID="a5aa1181d9410ceee46ac93620b83b05f94da18353ae93b457d758ac7ba023f3"
Nov 02 08:45:00.563190 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.563140    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:45:00.688092 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688053    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-certs\") pod \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") "
Nov 02 08:45:00.688336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688109    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-socket\") pod \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") "
Nov 02 08:45:00.688336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688162    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-podinfo\") pod \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") "
Nov 02 08:45:00.688336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688221    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-data\") pod \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") "
Nov 02 08:45:00.688336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688269    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-token\" (UniqueName: \"kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-token\") pod \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") "
Nov 02 08:45:00.688336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688303    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-credential-socket\") pod \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") "
Nov 02 08:45:00.688635 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688344    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istiod-ca-cert\") pod \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") "
Nov 02 08:45:00.688635 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688367    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-certs" pod "62f70659-a519-4f21-b29d-4e8ba2b4cf66" (UID: "62f70659-a519-4f21-b29d-4e8ba2b4cf66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:45:00.688635 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688444    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-envoy\") pod \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") "
Nov 02 08:45:00.688635 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688518    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-kube-api-access-jqrbr\" (UniqueName: \"kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-kube-api-access-jqrbr\") pod \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\" (UID: \"62f70659-a519-4f21-b29d-4e8ba2b4cf66\") "
Nov 02 08:45:00.688635 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688444    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-socket" pod "62f70659-a519-4f21-b29d-4e8ba2b4cf66" (UID: "62f70659-a519-4f21-b29d-4e8ba2b4cf66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:45:00.688971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688678    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-data" pod "62f70659-a519-4f21-b29d-4e8ba2b4cf66" (UID: "62f70659-a519-4f21-b29d-4e8ba2b4cf66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:45:00.688971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688695    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-credential-socket" pod "62f70659-a519-4f21-b29d-4e8ba2b4cf66" (UID: "62f70659-a519-4f21-b29d-4e8ba2b4cf66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:45:00.688971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688937    4898 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-certs\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:45:00.688971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688960    4898 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-workload-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:45:00.688971 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688973    4898 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-data\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:45:00.689187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688988    4898 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-credential-socket\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:45:00.689187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.688998    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istiod-ca-cert" pod "62f70659-a519-4f21-b29d-4e8ba2b4cf66" (UID: "62f70659-a519-4f21-b29d-4e8ba2b4cf66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:45:00.691101 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.691075    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-envoy" pod "62f70659-a519-4f21-b29d-4e8ba2b4cf66" (UID: "62f70659-a519-4f21-b29d-4e8ba2b4cf66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:45:00.691209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.691111    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-token" pod "62f70659-a519-4f21-b29d-4e8ba2b4cf66" (UID: "62f70659-a519-4f21-b29d-4e8ba2b4cf66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:45:00.691209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.691125    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-podinfo" pod "62f70659-a519-4f21-b29d-4e8ba2b4cf66" (UID: "62f70659-a519-4f21-b29d-4e8ba2b4cf66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 08:45:00.691209 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.691161    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-kube-api-access-jqrbr" pod "62f70659-a519-4f21-b29d-4e8ba2b4cf66" (UID: "62f70659-a519-4f21-b29d-4e8ba2b4cf66"). InnerVolumeSpecName "kube-api-access-jqrbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:45:00.789881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.789845    4898 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-token\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:45:00.789881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.789877    4898 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istiod-ca-cert\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:45:00.790114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.789903    4898 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-envoy\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:45:00.790114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.789917    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqrbr\" (UniqueName: \"kubernetes.io/projected/62f70659-a519-4f21-b29d-4e8ba2b4cf66-kube-api-access-jqrbr\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:45:00.790114 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:00.789932    4898 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62f70659-a519-4f21-b29d-4e8ba2b4cf66-istio-podinfo\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:45:01.528670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:01.528637    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf" event={"ID":"62f70659-a519-4f21-b29d-4e8ba2b4cf66","Type":"ContainerDied","Data":"00e00d52a961ec810f5e93b1206dcee333618751351e82127833bc81f5f7bc3d"}
Nov 02 08:45:01.528670 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:01.528667    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"
Nov 02 08:45:01.529232 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:01.528709    4898 scope.go:122] "RemoveContainer" containerID="9087ad5cd0e66d0b2bd2b72b9eec579a37c88923776d067a6fb7e86d51bc5620"
Nov 02 08:45:01.596238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:01.596202    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"]
Nov 02 08:45:01.608696 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:01.608663    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1b623b-openshift-default-55946d8765cz8kf"]
Nov 02 08:45:03.463139 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:03.463101    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" path="/var/lib/kubelet/pods/62f70659-a519-4f21-b29d-4e8ba2b4cf66/volumes"
Nov 02 08:45:07.022042 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:07.022008    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jxxsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:45:07.022518 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:07.022494    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm" podUID="12755c83-2699-430b-8342-9c06b8273215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:45:14.987935 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:45:14.987874    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:45:22.024081 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:22.024034    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jxxsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:45:22.024772 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:22.024718    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jxxsm" podUID="12755c83-2699-430b-8342-9c06b8273215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:45:38.128573 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:38.128526    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:45:38.129449 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:38.129410    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:45:44.988870 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:45:44.988825    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:45:53.128189 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:53.128058    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:45:53.129037 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:45:53.128999    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:46:14.988497 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:46:14.988450    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:46:44.988217 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:46:44.988177    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:47:14.993383 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:47:14.993332    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:47:44.988572 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:47:44.988528    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:48:14.989759 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:48:14.989719    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:48:44.031239 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:48:44.031194    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"]
Nov 02 08:48:44.043211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:48:44.043181    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8966ac03-sjnxt"]
Nov 02 08:48:44.988457 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:48:44.988416    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:48:45.463189 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:48:45.463157    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85827207-d34c-472d-af30-2bc2182f52ff" path="/var/lib/kubelet/pods/85827207-d34c-472d-af30-2bc2182f52ff/volumes"
Nov 02 08:49:10.980258 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:49:10.980164    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:49:10.993374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:49:10.993351    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:49:14.987971 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:49:14.987919    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:49:44.987845 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:49:44.987805    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:50:14.988221 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:50:14.988171    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:50:16.032667 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:50:16.032632    4898 scope.go:122] "RemoveContainer" containerID="cbb3c67376b4fceaf8632784e8f1f5e37a014753099942efdffbcbd443af891d"
Nov 02 08:50:44.989284 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:50:44.989239    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:51:15.014768 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:51:15.014722    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:51:44.988308 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:51:44.988271    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:52:14.989402 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:52:14.989314    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:52:44.988224 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:52:44.988183    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:53:14.991958 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:53:14.991824    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:53:44.987503 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:53:44.987449    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:54:11.016374 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:54:11.016346    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:54:11.032974 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:54:11.032952    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:54:14.990261 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:54:14.990220    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:54:44.987597 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:54:44.987557    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:55:14.987871 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:55:14.987784    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:55:37.036463 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.036420    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"]
Nov 02 08:55:37.037006 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.036660    4898 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk" podUID="d2c780cc-30a6-43ed-afe5-77b774bb026f" containerName="manager" containerID="cri-o://35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb" gracePeriod=10
Nov 02 08:55:37.289678 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.289617    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"
Nov 02 08:55:37.372223 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.372186    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c780cc-30a6-43ed-afe5-77b774bb026f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2c780cc-30a6-43ed-afe5-77b774bb026f-extensions-socket-volume\") pod \"d2c780cc-30a6-43ed-afe5-77b774bb026f\" (UID: \"d2c780cc-30a6-43ed-afe5-77b774bb026f\") "
Nov 02 08:55:37.372410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.372244    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c780cc-30a6-43ed-afe5-77b774bb026f-kube-api-access-9fsnv\" (UniqueName: \"kubernetes.io/projected/d2c780cc-30a6-43ed-afe5-77b774bb026f-kube-api-access-9fsnv\") pod \"d2c780cc-30a6-43ed-afe5-77b774bb026f\" (UID: \"d2c780cc-30a6-43ed-afe5-77b774bb026f\") "
Nov 02 08:55:37.372705 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.372662    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c780cc-30a6-43ed-afe5-77b774bb026f-extensions-socket-volume" pod "d2c780cc-30a6-43ed-afe5-77b774bb026f" (UID: "d2c780cc-30a6-43ed-afe5-77b774bb026f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 08:55:37.374575 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.374547    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c780cc-30a6-43ed-afe5-77b774bb026f-kube-api-access-9fsnv" pod "d2c780cc-30a6-43ed-afe5-77b774bb026f" (UID: "d2c780cc-30a6-43ed-afe5-77b774bb026f"). InnerVolumeSpecName "kube-api-access-9fsnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:55:37.470683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.470555    4898 generic.go:356] "Generic (PLEG): container finished" podID="d2c780cc-30a6-43ed-afe5-77b774bb026f" containerID="35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb" exitCode=0
Nov 02 08:55:37.470683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.470607    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk" event={"ID":"d2c780cc-30a6-43ed-afe5-77b774bb026f","Type":"ContainerDied","Data":"35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb"}
Nov 02 08:55:37.470683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.470622    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"
Nov 02 08:55:37.470683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.470649    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk" event={"ID":"d2c780cc-30a6-43ed-afe5-77b774bb026f","Type":"ContainerDied","Data":"d740e8f48d51177d8f46dceb32a60e55890871b467b596b9b2077105adb87bb8"}
Nov 02 08:55:37.470683 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.470679    4898 scope.go:122] "RemoveContainer" containerID="35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb"
Nov 02 08:55:37.473224 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.473199    4898 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2c780cc-30a6-43ed-afe5-77b774bb026f-extensions-socket-volume\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:55:37.473224 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.473223    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fsnv\" (UniqueName: \"kubernetes.io/projected/d2c780cc-30a6-43ed-afe5-77b774bb026f-kube-api-access-9fsnv\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:55:37.486303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.486284    4898 scope.go:122] "RemoveContainer" containerID="35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb"
Nov 02 08:55:37.486580 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:55:37.486558    4898 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb\": container with ID starting with 35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb not found: ID does not exist" containerID="35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb"
Nov 02 08:55:37.486640 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.486588    4898 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb"} err="failed to get container status \"35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb\": rpc error: code = NotFound desc = could not find container \"35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb\": container with ID starting with 35adc54b93fbcde68f52857a826288fee011295a58d0fb24a38cf796e0f848bb not found: ID does not exist"
Nov 02 08:55:37.523002 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.522973    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"]
Nov 02 08:55:37.526620 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:37.526598    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-br6sk"]
Nov 02 08:55:39.462671 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:55:39.462636    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c780cc-30a6-43ed-afe5-77b774bb026f" path="/var/lib/kubelet/pods/d2c780cc-30a6-43ed-afe5-77b774bb026f/volumes"
Nov 02 08:55:44.987555 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:55:44.987514    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:56:14.988247 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:56:14.988205    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:56:40.186315 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186277    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"]
Nov 02 08:56:40.186781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186707    4898 cpu_manager.go:414] "containerMap: removing container" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy"
Nov 02 08:56:40.186781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186719    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy"
Nov 02 08:56:40.186781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186742    4898 cpu_manager.go:414] "containerMap: removing container" podUID="eb904c60-e123-45df-b8ef-b44a83dc12a7" containerName="payload-processing"
Nov 02 08:56:40.186781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186747    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb904c60-e123-45df-b8ef-b44a83dc12a7" containerName="payload-processing"
Nov 02 08:56:40.186781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186768    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1f3fb475-4d09-4a86-80ab-10e6d38a9163" containerName="payload-pre-processing"
Nov 02 08:56:40.186781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186776    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3fb475-4d09-4a86-80ab-10e6d38a9163" containerName="payload-pre-processing"
Nov 02 08:56:40.187030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186794    4898 cpu_manager.go:414] "containerMap: removing container" podUID="85827207-d34c-472d-af30-2bc2182f52ff" containerName="revoke-keys"
Nov 02 08:56:40.187030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186799    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85827207-d34c-472d-af30-2bc2182f52ff" containerName="revoke-keys"
Nov 02 08:56:40.187030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186807    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d2c780cc-30a6-43ed-afe5-77b774bb026f" containerName="manager"
Nov 02 08:56:40.187030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186812    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c780cc-30a6-43ed-afe5-77b774bb026f" containerName="manager"
Nov 02 08:56:40.187030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186880    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy"
Nov 02 08:56:40.187030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186903    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb904c60-e123-45df-b8ef-b44a83dc12a7" containerName="payload-processing"
Nov 02 08:56:40.187030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186912    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3fb475-4d09-4a86-80ab-10e6d38a9163" containerName="payload-pre-processing"
Nov 02 08:56:40.187030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186922    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="85827207-d34c-472d-af30-2bc2182f52ff" containerName="revoke-keys"
Nov 02 08:56:40.187030 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.186932    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c780cc-30a6-43ed-afe5-77b774bb026f" containerName="manager"
Nov 02 08:56:40.192375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.192341    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"
Nov 02 08:56:40.206100 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.206071    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"]
Nov 02 08:56:40.216106 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.216081    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hnnz8\""
Nov 02 08:56:40.292457 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.292418    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f96911bb-0980-48ba-8c8d-f97c5e897bac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-w95pb\" (UID: \"f96911bb-0980-48ba-8c8d-f97c5e897bac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"
Nov 02 08:56:40.292647 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.292526    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8cd\" (UniqueName: \"kubernetes.io/projected/f96911bb-0980-48ba-8c8d-f97c5e897bac-kube-api-access-vv8cd\") pod \"kuadrant-operator-controller-manager-678d87766f-w95pb\" (UID: \"f96911bb-0980-48ba-8c8d-f97c5e897bac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"
Nov 02 08:56:40.393199 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.393162    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f96911bb-0980-48ba-8c8d-f97c5e897bac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-w95pb\" (UID: \"f96911bb-0980-48ba-8c8d-f97c5e897bac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"
Nov 02 08:56:40.393410 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.393230    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8cd\" (UniqueName: \"kubernetes.io/projected/f96911bb-0980-48ba-8c8d-f97c5e897bac-kube-api-access-vv8cd\") pod \"kuadrant-operator-controller-manager-678d87766f-w95pb\" (UID: \"f96911bb-0980-48ba-8c8d-f97c5e897bac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"
Nov 02 08:56:40.393591 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.393570    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f96911bb-0980-48ba-8c8d-f97c5e897bac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-w95pb\" (UID: \"f96911bb-0980-48ba-8c8d-f97c5e897bac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"
Nov 02 08:56:40.420719 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.420692    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8cd\" (UniqueName: \"kubernetes.io/projected/f96911bb-0980-48ba-8c8d-f97c5e897bac-kube-api-access-vv8cd\") pod \"kuadrant-operator-controller-manager-678d87766f-w95pb\" (UID: \"f96911bb-0980-48ba-8c8d-f97c5e897bac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"
Nov 02 08:56:40.508470 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.508381    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"
Nov 02 08:56:40.844340 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:40.844258    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"]
Nov 02 08:56:40.857249 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:56:40.857222    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96911bb_0980_48ba_8c8d_f97c5e897bac.slice/crio-35beaac1a173520d1cf59d3a7ab4bbe3e6fb6f503c94063f794e045a38fa050d WatchSource:0}: Error finding container 35beaac1a173520d1cf59d3a7ab4bbe3e6fb6f503c94063f794e045a38fa050d: Status 404 returned error can't find the container with id 35beaac1a173520d1cf59d3a7ab4bbe3e6fb6f503c94063f794e045a38fa050d
Nov 02 08:56:41.822800 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:41.822761    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb" event={"ID":"f96911bb-0980-48ba-8c8d-f97c5e897bac","Type":"ContainerStarted","Data":"7d4f1a6cddfa5fad80d38e744f99d2c6e1aa4bd32d706a7d4d5fc25ab19cdbb9"}
Nov 02 08:56:41.823275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:41.822822    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb" event={"ID":"f96911bb-0980-48ba-8c8d-f97c5e897bac","Type":"ContainerStarted","Data":"35beaac1a173520d1cf59d3a7ab4bbe3e6fb6f503c94063f794e045a38fa050d"}
Nov 02 08:56:41.823275 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:41.822844    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"
Nov 02 08:56:41.896490 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:41.896432    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb" podStartSLOduration=1.8964170839999999 podStartE2EDuration="1.896417084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:56:41.882736863 +0000 UTC m=+5552.573063847" watchObservedRunningTime="2026-11-02 08:56:41.896417084 +0000 UTC m=+5552.586744050"
Nov 02 08:56:44.988850 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:56:44.988808    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:56:52.830861 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:56:52.830825    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w95pb"
Nov 02 08:57:14.993849 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:57:14.993802    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:57:45.029069 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:45.029014    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnq5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:57:45.029863 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:45.029820    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" podUID="dea9d7e7-103a-4e9a-8824-e62a72191cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:57:45.048642 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:57:45.048601    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:57:49.937336 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:49.937300    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"]
Nov 02 08:57:49.937769 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:49.937744    4898 cpu_manager.go:414] "containerMap: removing container" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy"
Nov 02 08:57:49.937769 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:49.937755    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy"
Nov 02 08:57:49.937853 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:49.937820    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f70659-a519-4f21-b29d-4e8ba2b4cf66" containerName="istio-proxy"
Nov 02 08:57:49.941314 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:49.941296    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:49.969326 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:49.969248    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:57:49.969326 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:49.969283    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 08:57:49.969326 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:49.969290    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:57:49.969326 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:49.969327    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:57:49.971725 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:49.971694    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"]
Nov 02 08:57:50.027454 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:50.027417    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f52945-kqlnv\" (UID: \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:50.027654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:50.027476    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kgt7\" (UniqueName: \"kubernetes.io/projected/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-kube-api-access-7kgt7\") pod \"maas-api-revoke-sub-43f52945-kqlnv\" (UID: \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:50.027654 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:50.027581    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f52945-kqlnv\" (UID: \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:50.128238 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:50.128196    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f52945-kqlnv\" (UID: \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:50.128434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:50.128276    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kgt7\" (UniqueName: \"kubernetes.io/projected/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-kube-api-access-7kgt7\") pod \"maas-api-revoke-sub-43f52945-kqlnv\" (UID: \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:50.128434 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:50.128309    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f52945-kqlnv\" (UID: \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:50.128946 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:50.128923    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f52945-kqlnv\" (UID: \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:50.130923 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:50.130900    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f52945-kqlnv\" (UID: \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:50.155334 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:50.155295    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kgt7\" (UniqueName: \"kubernetes.io/projected/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-kube-api-access-7kgt7\") pod \"maas-api-revoke-sub-43f52945-kqlnv\" (UID: \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:50.254608 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:50.254526    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:50.570531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:50.570498    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"]
Nov 02 08:57:50.572311 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:57:50.572278    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b6e7de_1a72_4ce2_b6d9_c6149a69e845.slice/crio-7a5874612a2dd1fda009ca91ff5d7d340c346f19ad0f4e76f0ee9c86353a81ee WatchSource:0}: Error finding container 7a5874612a2dd1fda009ca91ff5d7d340c346f19ad0f4e76f0ee9c86353a81ee: Status 404 returned error can't find the container with id 7a5874612a2dd1fda009ca91ff5d7d340c346f19ad0f4e76f0ee9c86353a81ee
Nov 02 08:57:51.323057 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:51.323010    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv" event={"ID":"79b6e7de-1a72-4ce2-b6d9-c6149a69e845","Type":"ContainerStarted","Data":"abaadaca095e5357cdfd9842e41f9b32b82453b1f15d544301c287e7fd8fcc92"}
Nov 02 08:57:51.323487 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:51.323069    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv" event={"ID":"79b6e7de-1a72-4ce2-b6d9-c6149a69e845","Type":"ContainerStarted","Data":"7a5874612a2dd1fda009ca91ff5d7d340c346f19ad0f4e76f0ee9c86353a81ee"}
Nov 02 08:57:51.365453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:51.365392    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv" podStartSLOduration=2.365376574 podStartE2EDuration="2.365376574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:57:51.363218536 +0000 UTC m=+5622.053545501" watchObservedRunningTime="2026-11-02 08:57:51.365376574 +0000 UTC m=+5622.055703568"
Nov 02 08:57:52.330311 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:52.330273    4898 generic.go:356] "Generic (PLEG): container finished" podID="79b6e7de-1a72-4ce2-b6d9-c6149a69e845" containerID="abaadaca095e5357cdfd9842e41f9b32b82453b1f15d544301c287e7fd8fcc92" exitCode=0
Nov 02 08:57:52.330764 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:52.330364    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv" event={"ID":"79b6e7de-1a72-4ce2-b6d9-c6149a69e845","Type":"ContainerDied","Data":"abaadaca095e5357cdfd9842e41f9b32b82453b1f15d544301c287e7fd8fcc92"}
Nov 02 08:57:53.485338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:53.485308    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:53.559061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:53.559021    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-kube-api-access-7kgt7\" (UniqueName: \"kubernetes.io/projected/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-kube-api-access-7kgt7\") pod \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\" (UID: \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\") "
Nov 02 08:57:53.559061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:53.559068    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-openshift-ca\") pod \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\" (UID: \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\") "
Nov 02 08:57:53.559334 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:53.559137    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-xks-ca\") pod \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\" (UID: \"79b6e7de-1a72-4ce2-b6d9-c6149a69e845\") "
Nov 02 08:57:53.559562 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:53.559531    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-openshift-ca" pod "79b6e7de-1a72-4ce2-b6d9-c6149a69e845" (UID: "79b6e7de-1a72-4ce2-b6d9-c6149a69e845"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:57:53.561233 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:53.561206    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-xks-ca" pod "79b6e7de-1a72-4ce2-b6d9-c6149a69e845" (UID: "79b6e7de-1a72-4ce2-b6d9-c6149a69e845"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:57:53.561375 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:53.561352    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-kube-api-access-7kgt7" pod "79b6e7de-1a72-4ce2-b6d9-c6149a69e845" (UID: "79b6e7de-1a72-4ce2-b6d9-c6149a69e845"). InnerVolumeSpecName "kube-api-access-7kgt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:57:53.660288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:53.660254    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:57:53.660288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:53.660287    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kgt7\" (UniqueName: \"kubernetes.io/projected/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-kube-api-access-7kgt7\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:57:53.660491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:53.660299    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b6e7de-1a72-4ce2-b6d9-c6149a69e845-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:57:54.343351 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:54.343313    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"
Nov 02 08:57:54.343546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:54.343343    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv" event={"ID":"79b6e7de-1a72-4ce2-b6d9-c6149a69e845","Type":"ContainerDied","Data":"7a5874612a2dd1fda009ca91ff5d7d340c346f19ad0f4e76f0ee9c86353a81ee"}
Nov 02 08:57:54.343546 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:57:54.343397    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a5874612a2dd1fda009ca91ff5d7d340c346f19ad0f4e76f0ee9c86353a81ee"
Nov 02 08:58:00.278685 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.278645    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"]
Nov 02 08:58:00.279174 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.279158    4898 cpu_manager.go:414] "containerMap: removing container" podUID="79b6e7de-1a72-4ce2-b6d9-c6149a69e845" containerName="revoke-keys"
Nov 02 08:58:00.279234 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.279176    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b6e7de-1a72-4ce2-b6d9-c6149a69e845" containerName="revoke-keys"
Nov 02 08:58:00.279288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.279277    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b6e7de-1a72-4ce2-b6d9-c6149a69e845" containerName="revoke-keys"
Nov 02 08:58:00.282360 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.282340    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:00.310004 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.309962    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:58:00.310181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.310022    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 08:58:00.312749 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.312719    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"]
Nov 02 08:58:00.324024 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.323994    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0567b209-rhdph\" (UID: \"f54ae926-f014-4013-9c9c-9b768ac99559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:00.324188 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.324095    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0567b209-rhdph\" (UID: \"f54ae926-f014-4013-9c9c-9b768ac99559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:00.324259 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.324183    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcphq\" (UniqueName: \"kubernetes.io/projected/f54ae926-f014-4013-9c9c-9b768ac99559-kube-api-access-tcphq\") pod \"maas-api-revoke-sub-0567b209-rhdph\" (UID: \"f54ae926-f014-4013-9c9c-9b768ac99559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:00.358502 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.358466    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:58:00.358502 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.358466    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:58:00.425156 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.425115    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcphq\" (UniqueName: \"kubernetes.io/projected/f54ae926-f014-4013-9c9c-9b768ac99559-kube-api-access-tcphq\") pod \"maas-api-revoke-sub-0567b209-rhdph\" (UID: \"f54ae926-f014-4013-9c9c-9b768ac99559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:00.425346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.425181    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0567b209-rhdph\" (UID: \"f54ae926-f014-4013-9c9c-9b768ac99559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:00.425346 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.425245    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0567b209-rhdph\" (UID: \"f54ae926-f014-4013-9c9c-9b768ac99559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:00.426029 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.425994    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0567b209-rhdph\" (UID: \"f54ae926-f014-4013-9c9c-9b768ac99559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:00.427710 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.427688    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0567b209-rhdph\" (UID: \"f54ae926-f014-4013-9c9c-9b768ac99559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:00.448531 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.448501    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcphq\" (UniqueName: \"kubernetes.io/projected/f54ae926-f014-4013-9c9c-9b768ac99559-kube-api-access-tcphq\") pod \"maas-api-revoke-sub-0567b209-rhdph\" (UID: \"f54ae926-f014-4013-9c9c-9b768ac99559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:00.595805 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.595721    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:00.896858 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:00.896714    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"]
Nov 02 08:58:00.900216 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:58:00.900181    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54ae926_f014_4013_9c9c_9b768ac99559.slice/crio-42b4396f4759c7109c9a03ec3489c2c1e0f28b6a26b11aa9fefa79d675cdc061 WatchSource:0}: Error finding container 42b4396f4759c7109c9a03ec3489c2c1e0f28b6a26b11aa9fefa79d675cdc061: Status 404 returned error can't find the container with id 42b4396f4759c7109c9a03ec3489c2c1e0f28b6a26b11aa9fefa79d675cdc061
Nov 02 08:58:01.383093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:01.383050    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph" event={"ID":"f54ae926-f014-4013-9c9c-9b768ac99559","Type":"ContainerStarted","Data":"0a61029dcd57c55b365b13dbdd9b389c59a5637e1252b7b0c75e91b24f4b2c44"}
Nov 02 08:58:01.383589 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:01.383121    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph" event={"ID":"f54ae926-f014-4013-9c9c-9b768ac99559","Type":"ContainerStarted","Data":"42b4396f4759c7109c9a03ec3489c2c1e0f28b6a26b11aa9fefa79d675cdc061"}
Nov 02 08:58:01.425476 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:01.425374    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph" podStartSLOduration=1.425360569 podStartE2EDuration="1.425360569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:58:01.42358266 +0000 UTC m=+5632.113909626" watchObservedRunningTime="2026-11-02 08:58:01.425360569 +0000 UTC m=+5632.115687532"
Nov 02 08:58:02.391083 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:02.391043    4898 generic.go:356] "Generic (PLEG): container finished" podID="f54ae926-f014-4013-9c9c-9b768ac99559" containerID="0a61029dcd57c55b365b13dbdd9b389c59a5637e1252b7b0c75e91b24f4b2c44" exitCode=0
Nov 02 08:58:02.391519 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:02.391118    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph" event={"ID":"f54ae926-f014-4013-9c9c-9b768ac99559","Type":"ContainerDied","Data":"0a61029dcd57c55b365b13dbdd9b389c59a5637e1252b7b0c75e91b24f4b2c44"}
Nov 02 08:58:03.539550 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:03.539519    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:03.658746 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:03.658712    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-xks-ca\") pod \"f54ae926-f014-4013-9c9c-9b768ac99559\" (UID: \"f54ae926-f014-4013-9c9c-9b768ac99559\") "
Nov 02 08:58:03.658962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:03.658761    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-openshift-ca\") pod \"f54ae926-f014-4013-9c9c-9b768ac99559\" (UID: \"f54ae926-f014-4013-9c9c-9b768ac99559\") "
Nov 02 08:58:03.658962 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:03.658918    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54ae926-f014-4013-9c9c-9b768ac99559-kube-api-access-tcphq\" (UniqueName: \"kubernetes.io/projected/f54ae926-f014-4013-9c9c-9b768ac99559-kube-api-access-tcphq\") pod \"f54ae926-f014-4013-9c9c-9b768ac99559\" (UID: \"f54ae926-f014-4013-9c9c-9b768ac99559\") "
Nov 02 08:58:03.659232 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:03.659203    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-openshift-ca" pod "f54ae926-f014-4013-9c9c-9b768ac99559" (UID: "f54ae926-f014-4013-9c9c-9b768ac99559"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:58:03.661093 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:03.661070    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-xks-ca" pod "f54ae926-f014-4013-9c9c-9b768ac99559" (UID: "f54ae926-f014-4013-9c9c-9b768ac99559"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:58:03.661244 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:03.661223    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54ae926-f014-4013-9c9c-9b768ac99559-kube-api-access-tcphq" pod "f54ae926-f014-4013-9c9c-9b768ac99559" (UID: "f54ae926-f014-4013-9c9c-9b768ac99559"). InnerVolumeSpecName "kube-api-access-tcphq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:58:03.760333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:03.760294    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcphq\" (UniqueName: \"kubernetes.io/projected/f54ae926-f014-4013-9c9c-9b768ac99559-kube-api-access-tcphq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:58:03.760333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:03.760330    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:58:03.760333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:03.760341    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54ae926-f014-4013-9c9c-9b768ac99559-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:58:04.403781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:04.403752    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"
Nov 02 08:58:04.403993 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:04.403785    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph" event={"ID":"f54ae926-f014-4013-9c9c-9b768ac99559","Type":"ContainerDied","Data":"42b4396f4759c7109c9a03ec3489c2c1e0f28b6a26b11aa9fefa79d675cdc061"}
Nov 02 08:58:04.403993 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:04.403830    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42b4396f4759c7109c9a03ec3489c2c1e0f28b6a26b11aa9fefa79d675cdc061"
Nov 02 08:58:14.988681 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:58:14.988642    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:58:22.598947 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.598881    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"]
Nov 02 08:58:22.601491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.599521    4898 cpu_manager.go:414] "containerMap: removing container" podUID="f54ae926-f014-4013-9c9c-9b768ac99559" containerName="revoke-keys"
Nov 02 08:58:22.601491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.599540    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54ae926-f014-4013-9c9c-9b768ac99559" containerName="revoke-keys"
Nov 02 08:58:22.601491 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.599671    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54ae926-f014-4013-9c9c-9b768ac99559" containerName="revoke-keys"
Nov 02 08:58:22.603143 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.603120    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:22.619433 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.619403    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:58:22.631695 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.631657    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:58:22.631695 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.631659    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:58:22.631991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.631748    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 08:58:22.634742 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.634714    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"]
Nov 02 08:58:22.678246 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.678206    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg7kp\" (UniqueName: \"kubernetes.io/projected/d460dfbb-6535-4108-9c62-23de05999e7a-kube-api-access-xg7kp\") pod \"maas-api-revoke-sub-ed15ac12-wsc7m\" (UID: \"d460dfbb-6535-4108-9c62-23de05999e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:22.678419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.678248    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed15ac12-wsc7m\" (UID: \"d460dfbb-6535-4108-9c62-23de05999e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:22.678419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.678356    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed15ac12-wsc7m\" (UID: \"d460dfbb-6535-4108-9c62-23de05999e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:22.779352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.779311    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg7kp\" (UniqueName: \"kubernetes.io/projected/d460dfbb-6535-4108-9c62-23de05999e7a-kube-api-access-xg7kp\") pod \"maas-api-revoke-sub-ed15ac12-wsc7m\" (UID: \"d460dfbb-6535-4108-9c62-23de05999e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:22.779352 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.779353    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed15ac12-wsc7m\" (UID: \"d460dfbb-6535-4108-9c62-23de05999e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:22.779587 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.779396    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed15ac12-wsc7m\" (UID: \"d460dfbb-6535-4108-9c62-23de05999e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:22.780213 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.780189    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed15ac12-wsc7m\" (UID: \"d460dfbb-6535-4108-9c62-23de05999e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:22.781872 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.781852    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed15ac12-wsc7m\" (UID: \"d460dfbb-6535-4108-9c62-23de05999e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:22.802844 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.802817    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg7kp\" (UniqueName: \"kubernetes.io/projected/d460dfbb-6535-4108-9c62-23de05999e7a-kube-api-access-xg7kp\") pod \"maas-api-revoke-sub-ed15ac12-wsc7m\" (UID: \"d460dfbb-6535-4108-9c62-23de05999e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:22.916419 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:22.916374    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:23.216211 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:23.216183    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"]
Nov 02 08:58:23.218221 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:58:23.218185    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd460dfbb_6535_4108_9c62_23de05999e7a.slice/crio-a5382a76a936f15386d161d2e09cf14bfaa8d9e2a55ef3929d411288569b1944 WatchSource:0}: Error finding container a5382a76a936f15386d161d2e09cf14bfaa8d9e2a55ef3929d411288569b1944: Status 404 returned error can't find the container with id a5382a76a936f15386d161d2e09cf14bfaa8d9e2a55ef3929d411288569b1944
Nov 02 08:58:23.563530 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:23.563493    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m" event={"ID":"d460dfbb-6535-4108-9c62-23de05999e7a","Type":"ContainerStarted","Data":"8d552ba42c3672644649e6f576c66e4cef587377cad7646c735c5672c0912f53"}
Nov 02 08:58:23.563728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:23.563560    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m" event={"ID":"d460dfbb-6535-4108-9c62-23de05999e7a","Type":"ContainerStarted","Data":"a5382a76a936f15386d161d2e09cf14bfaa8d9e2a55ef3929d411288569b1944"}
Nov 02 08:58:23.609905 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:23.609839    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m" podStartSLOduration=1.60982528 podStartE2EDuration="1.60982528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:58:23.606867874 +0000 UTC m=+5654.297194838" watchObservedRunningTime="2026-11-02 08:58:23.60982528 +0000 UTC m=+5654.300152241"
Nov 02 08:58:24.570938 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:24.570877    4898 generic.go:356] "Generic (PLEG): container finished" podID="d460dfbb-6535-4108-9c62-23de05999e7a" containerID="8d552ba42c3672644649e6f576c66e4cef587377cad7646c735c5672c0912f53" exitCode=0
Nov 02 08:58:24.571165 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:24.570963    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m" event={"ID":"d460dfbb-6535-4108-9c62-23de05999e7a","Type":"ContainerDied","Data":"8d552ba42c3672644649e6f576c66e4cef587377cad7646c735c5672c0912f53"}
Nov 02 08:58:25.717667 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:25.717641    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:25.809693 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:25.809659    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-openshift-ca\") pod \"d460dfbb-6535-4108-9c62-23de05999e7a\" (UID: \"d460dfbb-6535-4108-9c62-23de05999e7a\") "
Nov 02 08:58:25.809882 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:25.809854    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-xks-ca\") pod \"d460dfbb-6535-4108-9c62-23de05999e7a\" (UID: \"d460dfbb-6535-4108-9c62-23de05999e7a\") "
Nov 02 08:58:25.809991 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:25.809919    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d460dfbb-6535-4108-9c62-23de05999e7a-kube-api-access-xg7kp\" (UniqueName: \"kubernetes.io/projected/d460dfbb-6535-4108-9c62-23de05999e7a-kube-api-access-xg7kp\") pod \"d460dfbb-6535-4108-9c62-23de05999e7a\" (UID: \"d460dfbb-6535-4108-9c62-23de05999e7a\") "
Nov 02 08:58:25.810085 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:25.810061    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-openshift-ca" pod "d460dfbb-6535-4108-9c62-23de05999e7a" (UID: "d460dfbb-6535-4108-9c62-23de05999e7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:58:25.810239 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:25.810222    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:58:25.812065 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:25.812037    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-xks-ca" pod "d460dfbb-6535-4108-9c62-23de05999e7a" (UID: "d460dfbb-6535-4108-9c62-23de05999e7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:58:25.812193 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:25.812062    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d460dfbb-6535-4108-9c62-23de05999e7a-kube-api-access-xg7kp" pod "d460dfbb-6535-4108-9c62-23de05999e7a" (UID: "d460dfbb-6535-4108-9c62-23de05999e7a"). InnerVolumeSpecName "kube-api-access-xg7kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:58:25.911292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:25.911258    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d460dfbb-6535-4108-9c62-23de05999e7a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:58:25.911292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:25.911290    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg7kp\" (UniqueName: \"kubernetes.io/projected/d460dfbb-6535-4108-9c62-23de05999e7a-kube-api-access-xg7kp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:58:26.582141 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:26.582104    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m" event={"ID":"d460dfbb-6535-4108-9c62-23de05999e7a","Type":"ContainerDied","Data":"a5382a76a936f15386d161d2e09cf14bfaa8d9e2a55ef3929d411288569b1944"}
Nov 02 08:58:26.582333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:26.582160    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5382a76a936f15386d161d2e09cf14bfaa8d9e2a55ef3929d411288569b1944"
Nov 02 08:58:26.582333 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:26.582114    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"
Nov 02 08:58:45.023070 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:45.023020    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pnq5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 08:58:45.023729 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:45.023690    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pnq5t" podUID="dea9d7e7-103a-4e9a-8824-e62a72191cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 08:58:45.053170 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:58:45.053127    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:58:53.432127 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.432090    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"]
Nov 02 08:58:53.432704 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.432682    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d460dfbb-6535-4108-9c62-23de05999e7a" containerName="revoke-keys"
Nov 02 08:58:53.432752 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.432706    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d460dfbb-6535-4108-9c62-23de05999e7a" containerName="revoke-keys"
Nov 02 08:58:53.432808 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.432798    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d460dfbb-6535-4108-9c62-23de05999e7a" containerName="revoke-keys"
Nov 02 08:58:53.436497 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.436468    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:53.456210 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.456178    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"]
Nov 02 08:58:53.465642 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.465608    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:58:53.465843 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.465744    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:58:53.465956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.465837    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:58:53.465956 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.465852    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 08:58:53.549295 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.549255    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e649ed8-qtfvw\" (UID: \"3ea2338d-ff9c-4bde-920f-babc159d3a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:53.549509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.549418    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkmdb\" (UniqueName: \"kubernetes.io/projected/3ea2338d-ff9c-4bde-920f-babc159d3a05-kube-api-access-hkmdb\") pod \"maas-api-revoke-sub-6e649ed8-qtfvw\" (UID: \"3ea2338d-ff9c-4bde-920f-babc159d3a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:53.549509 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.549467    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e649ed8-qtfvw\" (UID: \"3ea2338d-ff9c-4bde-920f-babc159d3a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:53.650465 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.650425    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkmdb\" (UniqueName: \"kubernetes.io/projected/3ea2338d-ff9c-4bde-920f-babc159d3a05-kube-api-access-hkmdb\") pod \"maas-api-revoke-sub-6e649ed8-qtfvw\" (UID: \"3ea2338d-ff9c-4bde-920f-babc159d3a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:53.650728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.650470    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e649ed8-qtfvw\" (UID: \"3ea2338d-ff9c-4bde-920f-babc159d3a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:53.650728 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.650524    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e649ed8-qtfvw\" (UID: \"3ea2338d-ff9c-4bde-920f-babc159d3a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:53.651152 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.651131    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e649ed8-qtfvw\" (UID: \"3ea2338d-ff9c-4bde-920f-babc159d3a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:53.653098 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.653063    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e649ed8-qtfvw\" (UID: \"3ea2338d-ff9c-4bde-920f-babc159d3a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:53.674451 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.674427    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkmdb\" (UniqueName: \"kubernetes.io/projected/3ea2338d-ff9c-4bde-920f-babc159d3a05-kube-api-access-hkmdb\") pod \"maas-api-revoke-sub-6e649ed8-qtfvw\" (UID: \"3ea2338d-ff9c-4bde-920f-babc159d3a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:53.751453 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:53.751371    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:54.054415 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:54.054380    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"]
Nov 02 08:58:54.056046 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:58:54.056017    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea2338d_ff9c_4bde_920f_babc159d3a05.slice/crio-e150edb33012d50099196910197143a3d2849da306d4f2eee35fcbc878a0b0df WatchSource:0}: Error finding container e150edb33012d50099196910197143a3d2849da306d4f2eee35fcbc878a0b0df: Status 404 returned error can't find the container with id e150edb33012d50099196910197143a3d2849da306d4f2eee35fcbc878a0b0df
Nov 02 08:58:54.806965 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:54.806923    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw" event={"ID":"3ea2338d-ff9c-4bde-920f-babc159d3a05","Type":"ContainerStarted","Data":"d387b8a9c2beea4dc6b9febd4d61d55cc210026f1bccb7c18d66ff2484219740"}
Nov 02 08:58:54.807404 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:54.806981    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw" event={"ID":"3ea2338d-ff9c-4bde-920f-babc159d3a05","Type":"ContainerStarted","Data":"e150edb33012d50099196910197143a3d2849da306d4f2eee35fcbc878a0b0df"}
Nov 02 08:58:54.850338 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:54.850281    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw" podStartSLOduration=1.850268348 podStartE2EDuration="1.850268348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:58:54.847808304 +0000 UTC m=+5685.538135269" watchObservedRunningTime="2026-11-02 08:58:54.850268348 +0000 UTC m=+5685.540595312"
Nov 02 08:58:55.814580 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:55.814539    4898 generic.go:356] "Generic (PLEG): container finished" podID="3ea2338d-ff9c-4bde-920f-babc159d3a05" containerID="d387b8a9c2beea4dc6b9febd4d61d55cc210026f1bccb7c18d66ff2484219740" exitCode=0
Nov 02 08:58:55.815022 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:55.814627    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw" event={"ID":"3ea2338d-ff9c-4bde-920f-babc159d3a05","Type":"ContainerDied","Data":"d387b8a9c2beea4dc6b9febd4d61d55cc210026f1bccb7c18d66ff2484219740"}
Nov 02 08:58:56.965653 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:56.965627    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:57.083789 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:57.083704    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-xks-ca\") pod \"3ea2338d-ff9c-4bde-920f-babc159d3a05\" (UID: \"3ea2338d-ff9c-4bde-920f-babc159d3a05\") "
Nov 02 08:58:57.083789 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:57.083755    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea2338d-ff9c-4bde-920f-babc159d3a05-kube-api-access-hkmdb\" (UniqueName: \"kubernetes.io/projected/3ea2338d-ff9c-4bde-920f-babc159d3a05-kube-api-access-hkmdb\") pod \"3ea2338d-ff9c-4bde-920f-babc159d3a05\" (UID: \"3ea2338d-ff9c-4bde-920f-babc159d3a05\") "
Nov 02 08:58:57.083999 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:57.083800    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-openshift-ca\") pod \"3ea2338d-ff9c-4bde-920f-babc159d3a05\" (UID: \"3ea2338d-ff9c-4bde-920f-babc159d3a05\") "
Nov 02 08:58:57.084240 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:57.084203    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-openshift-ca" pod "3ea2338d-ff9c-4bde-920f-babc159d3a05" (UID: "3ea2338d-ff9c-4bde-920f-babc159d3a05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:58:57.085927 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:57.085902    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-xks-ca" pod "3ea2338d-ff9c-4bde-920f-babc159d3a05" (UID: "3ea2338d-ff9c-4bde-920f-babc159d3a05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:58:57.086041 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:57.085954    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea2338d-ff9c-4bde-920f-babc159d3a05-kube-api-access-hkmdb" pod "3ea2338d-ff9c-4bde-920f-babc159d3a05" (UID: "3ea2338d-ff9c-4bde-920f-babc159d3a05"). InnerVolumeSpecName "kube-api-access-hkmdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:58:57.185548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:57.185507    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:58:57.185548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:57.185543    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkmdb\" (UniqueName: \"kubernetes.io/projected/3ea2338d-ff9c-4bde-920f-babc159d3a05-kube-api-access-hkmdb\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:58:57.185548 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:57.185554    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2338d-ff9c-4bde-920f-babc159d3a05-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:58:57.828303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:57.828221    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"
Nov 02 08:58:57.828303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:57.828238    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw" event={"ID":"3ea2338d-ff9c-4bde-920f-babc159d3a05","Type":"ContainerDied","Data":"e150edb33012d50099196910197143a3d2849da306d4f2eee35fcbc878a0b0df"}
Nov 02 08:58:57.828303 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:58:57.828285    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e150edb33012d50099196910197143a3d2849da306d4f2eee35fcbc878a0b0df"
Nov 02 08:59:03.815878 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.815845    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"]
Nov 02 08:59:03.816300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.816286    4898 cpu_manager.go:414] "containerMap: removing container" podUID="3ea2338d-ff9c-4bde-920f-babc159d3a05" containerName="revoke-keys"
Nov 02 08:59:03.816300 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.816300    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea2338d-ff9c-4bde-920f-babc159d3a05" containerName="revoke-keys"
Nov 02 08:59:03.816405 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.816394    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea2338d-ff9c-4bde-920f-babc159d3a05" containerName="revoke-keys"
Nov 02 08:59:03.819605 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.819578    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:03.847668 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.847636    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:59:03.847836 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.847687    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 08:59:03.852061 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.852035    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"]
Nov 02 08:59:03.852996 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.852974    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:59:03.853095 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.853025    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:59:03.954820 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.954785    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461e1f5b-pxskt\" (UID: \"a6831a27-5834-454c-9edd-bd218a4d9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:03.955001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.954953    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgqpm\" (UniqueName: \"kubernetes.io/projected/a6831a27-5834-454c-9edd-bd218a4d9c8b-kube-api-access-pgqpm\") pod \"maas-api-revoke-sub-461e1f5b-pxskt\" (UID: \"a6831a27-5834-454c-9edd-bd218a4d9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:03.955001 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:03.954980    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461e1f5b-pxskt\" (UID: \"a6831a27-5834-454c-9edd-bd218a4d9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:04.055879 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:04.055838    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461e1f5b-pxskt\" (UID: \"a6831a27-5834-454c-9edd-bd218a4d9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:04.056097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:04.055965    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgqpm\" (UniqueName: \"kubernetes.io/projected/a6831a27-5834-454c-9edd-bd218a4d9c8b-kube-api-access-pgqpm\") pod \"maas-api-revoke-sub-461e1f5b-pxskt\" (UID: \"a6831a27-5834-454c-9edd-bd218a4d9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:04.056097 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:04.055988    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461e1f5b-pxskt\" (UID: \"a6831a27-5834-454c-9edd-bd218a4d9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:04.056530 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:04.056500    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461e1f5b-pxskt\" (UID: \"a6831a27-5834-454c-9edd-bd218a4d9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:04.058381 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:04.058359    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461e1f5b-pxskt\" (UID: \"a6831a27-5834-454c-9edd-bd218a4d9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:04.079463 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:04.079400    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgqpm\" (UniqueName: \"kubernetes.io/projected/a6831a27-5834-454c-9edd-bd218a4d9c8b-kube-api-access-pgqpm\") pod \"maas-api-revoke-sub-461e1f5b-pxskt\" (UID: \"a6831a27-5834-454c-9edd-bd218a4d9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:04.133487 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:04.133451    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:04.443556 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:04.443525    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"]
Nov 02 08:59:04.444975 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:59:04.444946    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6831a27_5834_454c_9edd_bd218a4d9c8b.slice/crio-45d4b73a9fb386350dc89e419cb5540ac80616cea36e3ef89068d6d256371e4a WatchSource:0}: Error finding container 45d4b73a9fb386350dc89e419cb5540ac80616cea36e3ef89068d6d256371e4a: Status 404 returned error can't find the container with id 45d4b73a9fb386350dc89e419cb5540ac80616cea36e3ef89068d6d256371e4a
Nov 02 08:59:04.870469 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:04.870429    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt" event={"ID":"a6831a27-5834-454c-9edd-bd218a4d9c8b","Type":"ContainerStarted","Data":"4ce93b943d4530739af5b9d2d3a15c460af3c9bf62bd3a1ec13affeb0f831997"}
Nov 02 08:59:04.870935 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:04.870482    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt" event={"ID":"a6831a27-5834-454c-9edd-bd218a4d9c8b","Type":"ContainerStarted","Data":"45d4b73a9fb386350dc89e419cb5540ac80616cea36e3ef89068d6d256371e4a"}
Nov 02 08:59:04.913288 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:04.913239    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt" podStartSLOduration=1.913225395 podStartE2EDuration="1.913225395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 08:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 08:59:04.910925575 +0000 UTC m=+5695.601252539" watchObservedRunningTime="2026-11-02 08:59:04.913225395 +0000 UTC m=+5695.603552358"
Nov 02 08:59:05.878074 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:05.878033    4898 generic.go:356] "Generic (PLEG): container finished" podID="a6831a27-5834-454c-9edd-bd218a4d9c8b" containerID="4ce93b943d4530739af5b9d2d3a15c460af3c9bf62bd3a1ec13affeb0f831997" exitCode=0
Nov 02 08:59:05.878511 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:05.878112    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt" event={"ID":"a6831a27-5834-454c-9edd-bd218a4d9c8b","Type":"ContainerDied","Data":"4ce93b943d4530739af5b9d2d3a15c460af3c9bf62bd3a1ec13affeb0f831997"}
Nov 02 08:59:07.033867 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.033834    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:07.186458 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.186418    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-openshift-ca\") pod \"a6831a27-5834-454c-9edd-bd218a4d9c8b\" (UID: \"a6831a27-5834-454c-9edd-bd218a4d9c8b\") "
Nov 02 08:59:07.186674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.186504    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-xks-ca\") pod \"a6831a27-5834-454c-9edd-bd218a4d9c8b\" (UID: \"a6831a27-5834-454c-9edd-bd218a4d9c8b\") "
Nov 02 08:59:07.186674 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.186543    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6831a27-5834-454c-9edd-bd218a4d9c8b-kube-api-access-pgqpm\" (UniqueName: \"kubernetes.io/projected/a6831a27-5834-454c-9edd-bd218a4d9c8b-kube-api-access-pgqpm\") pod \"a6831a27-5834-454c-9edd-bd218a4d9c8b\" (UID: \"a6831a27-5834-454c-9edd-bd218a4d9c8b\") "
Nov 02 08:59:07.186958 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.186923    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-openshift-ca" pod "a6831a27-5834-454c-9edd-bd218a4d9c8b" (UID: "a6831a27-5834-454c-9edd-bd218a4d9c8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:59:07.188745 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.188724    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6831a27-5834-454c-9edd-bd218a4d9c8b-kube-api-access-pgqpm" pod "a6831a27-5834-454c-9edd-bd218a4d9c8b" (UID: "a6831a27-5834-454c-9edd-bd218a4d9c8b"). InnerVolumeSpecName "kube-api-access-pgqpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:59:07.188833 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.188792    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-xks-ca" pod "a6831a27-5834-454c-9edd-bd218a4d9c8b" (UID: "a6831a27-5834-454c-9edd-bd218a4d9c8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:59:07.287756 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.287722    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:59:07.287756 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.287751    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6831a27-5834-454c-9edd-bd218a4d9c8b-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:59:07.287756 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.287761    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgqpm\" (UniqueName: \"kubernetes.io/projected/a6831a27-5834-454c-9edd-bd218a4d9c8b-kube-api-access-pgqpm\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:59:07.891450 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.891371    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt"
Nov 02 08:59:07.891450 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.891405    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461e1f5b-pxskt" event={"ID":"a6831a27-5834-454c-9edd-bd218a4d9c8b","Type":"ContainerDied","Data":"45d4b73a9fb386350dc89e419cb5540ac80616cea36e3ef89068d6d256371e4a"}
Nov 02 08:59:07.891649 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:07.891460    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45d4b73a9fb386350dc89e419cb5540ac80616cea36e3ef89068d6d256371e4a"
Nov 02 08:59:11.057878 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:11.057845    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:59:11.073714 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:11.073688    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 08:59:14.989956 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:59:14.989921    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:59:44.988318 ip-10-0-134-79 kubenswrapper[4898]: E1102 08:59:44.988279    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 08:59:45.036776 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.036742    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"]
Nov 02 08:59:45.037236 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.037220    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a6831a27-5834-454c-9edd-bd218a4d9c8b" containerName="revoke-keys"
Nov 02 08:59:45.037292 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.037237    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6831a27-5834-454c-9edd-bd218a4d9c8b" containerName="revoke-keys"
Nov 02 08:59:45.037331 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.037308    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6831a27-5834-454c-9edd-bd218a4d9c8b" containerName="revoke-keys"
Nov 02 08:59:45.040689 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.040671    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:45.068953 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.068924    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:59:45.069088 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.069007    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:59:45.069088 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.069020    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:59:45.069181 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.069123    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 08:59:45.073298 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.073270    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"]
Nov 02 08:59:45.156262 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.156229    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-653943bc-mbvxd\" (UID: \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:45.156420 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.156385    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-653943bc-mbvxd\" (UID: \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:45.156480 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.156433    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrhsd\" (UniqueName: \"kubernetes.io/projected/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-kube-api-access-rrhsd\") pod \"maas-api-revoke-sub-653943bc-mbvxd\" (UID: \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:45.257677 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.257614    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-653943bc-mbvxd\" (UID: \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:45.257677 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.257657    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrhsd\" (UniqueName: \"kubernetes.io/projected/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-kube-api-access-rrhsd\") pod \"maas-api-revoke-sub-653943bc-mbvxd\" (UID: \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:45.257840 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.257780    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-653943bc-mbvxd\" (UID: \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:45.258445 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.258423    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-653943bc-mbvxd\" (UID: \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:45.260050 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.260028    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-653943bc-mbvxd\" (UID: \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:45.282986 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.282960    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrhsd\" (UniqueName: \"kubernetes.io/projected/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-kube-api-access-rrhsd\") pod \"maas-api-revoke-sub-653943bc-mbvxd\" (UID: \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:45.353928 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.353875    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:45.653881 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:45.653847    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"]
Nov 02 08:59:45.655466 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:59:45.655428    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3fd186_ea0c_4961_ac89_47220cbd9b9c.slice/crio-a66f517d044ff19b4383c95668dbf8610048ecf13e8b63da19d9d6a932b5815a WatchSource:0}: Error finding container a66f517d044ff19b4383c95668dbf8610048ecf13e8b63da19d9d6a932b5815a: Status 404 returned error can't find the container with id a66f517d044ff19b4383c95668dbf8610048ecf13e8b63da19d9d6a932b5815a
Nov 02 08:59:46.173472 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:46.173437    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd" event={"ID":"ec3fd186-ea0c-4961-ac89-47220cbd9b9c","Type":"ContainerStarted","Data":"1c4d405f3ddd2025b968a1f6c835a113bd3487a3ebc357b0d83617019c24d81f"}
Nov 02 08:59:46.173857 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:46.173495    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd" event={"ID":"ec3fd186-ea0c-4961-ac89-47220cbd9b9c","Type":"ContainerStarted","Data":"a66f517d044ff19b4383c95668dbf8610048ecf13e8b63da19d9d6a932b5815a"}
Nov 02 08:59:47.181035 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:47.180999    4898 generic.go:356] "Generic (PLEG): container finished" podID="ec3fd186-ea0c-4961-ac89-47220cbd9b9c" containerID="1c4d405f3ddd2025b968a1f6c835a113bd3487a3ebc357b0d83617019c24d81f" exitCode=0
Nov 02 08:59:47.181478 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:47.181083    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd" event={"ID":"ec3fd186-ea0c-4961-ac89-47220cbd9b9c","Type":"ContainerDied","Data":"1c4d405f3ddd2025b968a1f6c835a113bd3487a3ebc357b0d83617019c24d81f"}
Nov 02 08:59:48.326023 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:48.325997    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:48.487533 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:48.487454    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-openshift-ca\") pod \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\" (UID: \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\") "
Nov 02 08:59:48.487533 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:48.487504    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-xks-ca\") pod \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\" (UID: \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\") "
Nov 02 08:59:48.487737 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:48.487659    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-kube-api-access-rrhsd\" (UniqueName: \"kubernetes.io/projected/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-kube-api-access-rrhsd\") pod \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\" (UID: \"ec3fd186-ea0c-4961-ac89-47220cbd9b9c\") "
Nov 02 08:59:48.487876 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:48.487848    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-openshift-ca" pod "ec3fd186-ea0c-4961-ac89-47220cbd9b9c" (UID: "ec3fd186-ea0c-4961-ac89-47220cbd9b9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:59:48.489680 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:48.489653    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-xks-ca" pod "ec3fd186-ea0c-4961-ac89-47220cbd9b9c" (UID: "ec3fd186-ea0c-4961-ac89-47220cbd9b9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:59:48.489818 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:48.489795    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-kube-api-access-rrhsd" pod "ec3fd186-ea0c-4961-ac89-47220cbd9b9c" (UID: "ec3fd186-ea0c-4961-ac89-47220cbd9b9c"). InnerVolumeSpecName "kube-api-access-rrhsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:59:48.589062 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:48.589030    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:59:48.589062 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:48.589058    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:59:48.589062 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:48.589067    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrhsd\" (UniqueName: \"kubernetes.io/projected/ec3fd186-ea0c-4961-ac89-47220cbd9b9c-kube-api-access-rrhsd\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:59:49.201699 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:49.201666    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd"
Nov 02 08:59:49.201944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:49.201705    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653943bc-mbvxd" event={"ID":"ec3fd186-ea0c-4961-ac89-47220cbd9b9c","Type":"ContainerDied","Data":"a66f517d044ff19b4383c95668dbf8610048ecf13e8b63da19d9d6a932b5815a"}
Nov 02 08:59:49.201944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:49.201749    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a66f517d044ff19b4383c95668dbf8610048ecf13e8b63da19d9d6a932b5815a"
Nov 02 08:59:55.392214 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.392177    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"]
Nov 02 08:59:55.392897 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.392868    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ec3fd186-ea0c-4961-ac89-47220cbd9b9c" containerName="revoke-keys"
Nov 02 08:59:55.392944 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.392902    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3fd186-ea0c-4961-ac89-47220cbd9b9c" containerName="revoke-keys"
Nov 02 08:59:55.393018 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.393006    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3fd186-ea0c-4961-ac89-47220cbd9b9c" containerName="revoke-keys"
Nov 02 08:59:55.396299 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.396280    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 08:59:55.416087 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.416055    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"]
Nov 02 08:59:55.423847 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.423818    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 08:59:55.424032 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.423877    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 08:59:55.449781 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.449751    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 08:59:55.449976 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.449838    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 08:59:55.551166 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.551129    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb887bca-rcsj4\" (UID: \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 08:59:55.551373 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.551203    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ps6\" (UniqueName: \"kubernetes.io/projected/bf81ff7f-b3b2-4821-abb5-258ed0bea849-kube-api-access-s7ps6\") pod \"maas-api-revoke-sub-eb887bca-rcsj4\" (UID: \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 08:59:55.551373 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.551303    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb887bca-rcsj4\" (UID: \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 08:59:55.652435 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.652400    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb887bca-rcsj4\" (UID: \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 08:59:55.652626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.652483    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ps6\" (UniqueName: \"kubernetes.io/projected/bf81ff7f-b3b2-4821-abb5-258ed0bea849-kube-api-access-s7ps6\") pod \"maas-api-revoke-sub-eb887bca-rcsj4\" (UID: \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 08:59:55.652626 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.652543    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb887bca-rcsj4\" (UID: \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 08:59:55.653305 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.653275    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb887bca-rcsj4\" (UID: \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 08:59:55.654992 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.654973    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb887bca-rcsj4\" (UID: \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 08:59:55.676287 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.676254    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ps6\" (UniqueName: \"kubernetes.io/projected/bf81ff7f-b3b2-4821-abb5-258ed0bea849-kube-api-access-s7ps6\") pod \"maas-api-revoke-sub-eb887bca-rcsj4\" (UID: \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 08:59:55.710009 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:55.709986    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 08:59:56.018633 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:56.018600    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"]
Nov 02 08:59:56.019317 ip-10-0-134-79 kubenswrapper[4898]: W1102 08:59:56.019291    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf81ff7f_b3b2_4821_abb5_258ed0bea849.slice/crio-6e2b30f261249eeda40a4035979017b753255fbc1c40855a6b3c37b963eaeb9a WatchSource:0}: Error finding container 6e2b30f261249eeda40a4035979017b753255fbc1c40855a6b3c37b963eaeb9a: Status 404 returned error can't find the container with id 6e2b30f261249eeda40a4035979017b753255fbc1c40855a6b3c37b963eaeb9a
Nov 02 08:59:56.254554 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:56.254517    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4" event={"ID":"bf81ff7f-b3b2-4821-abb5-258ed0bea849","Type":"ContainerStarted","Data":"6e2b30f261249eeda40a4035979017b753255fbc1c40855a6b3c37b963eaeb9a"}
Nov 02 08:59:57.264208 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:57.264109    4898 generic.go:356] "Generic (PLEG): container finished" podID="bf81ff7f-b3b2-4821-abb5-258ed0bea849" containerID="744995702d04fbb0e82cd5dea38a981292f451e52b2d2d3b03e1b5367d19e399" exitCode=0
Nov 02 08:59:57.264706 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:57.264207    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4" event={"ID":"bf81ff7f-b3b2-4821-abb5-258ed0bea849","Type":"ContainerDied","Data":"744995702d04fbb0e82cd5dea38a981292f451e52b2d2d3b03e1b5367d19e399"}
Nov 02 08:59:58.749957 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:58.747517    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 08:59:58.847492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:58.846314    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-xks-ca\") pod \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\" (UID: \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\") "
Nov 02 08:59:58.847492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:58.846374    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf81ff7f-b3b2-4821-abb5-258ed0bea849-kube-api-access-s7ps6\" (UniqueName: \"kubernetes.io/projected/bf81ff7f-b3b2-4821-abb5-258ed0bea849-kube-api-access-s7ps6\") pod \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\" (UID: \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\") "
Nov 02 08:59:58.847492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:58.846405    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-openshift-ca\") pod \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\" (UID: \"bf81ff7f-b3b2-4821-abb5-258ed0bea849\") "
Nov 02 08:59:58.847492 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:58.847141    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-openshift-ca" pod "bf81ff7f-b3b2-4821-abb5-258ed0bea849" (UID: "bf81ff7f-b3b2-4821-abb5-258ed0bea849"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 08:59:58.850203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:58.850171    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf81ff7f-b3b2-4821-abb5-258ed0bea849-kube-api-access-s7ps6" pod "bf81ff7f-b3b2-4821-abb5-258ed0bea849" (UID: "bf81ff7f-b3b2-4821-abb5-258ed0bea849"). InnerVolumeSpecName "kube-api-access-s7ps6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 08:59:58.850203 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:58.850186    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-xks-ca" pod "bf81ff7f-b3b2-4821-abb5-258ed0bea849" (UID: "bf81ff7f-b3b2-4821-abb5-258ed0bea849"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 08:59:58.948187 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:58.948085    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:59:58.948423 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:58.948402    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7ps6\" (UniqueName: \"kubernetes.io/projected/bf81ff7f-b3b2-4821-abb5-258ed0bea849-kube-api-access-s7ps6\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:59:58.948540 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:58.948523    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf81ff7f-b3b2-4821-abb5-258ed0bea849-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 08:59:59.309520 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:59.309426    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4" event={"ID":"bf81ff7f-b3b2-4821-abb5-258ed0bea849","Type":"ContainerDied","Data":"6e2b30f261249eeda40a4035979017b753255fbc1c40855a6b3c37b963eaeb9a"}
Nov 02 08:59:59.309766 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:59.309750    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e2b30f261249eeda40a4035979017b753255fbc1c40855a6b3c37b963eaeb9a"
Nov 02 08:59:59.309945 ip-10-0-134-79 kubenswrapper[4898]: I1102 08:59:59.309931    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb887bca-rcsj4"
Nov 02 09:00:00.137750 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.137676    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"]
Nov 02 09:00:00.138160 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.138113    4898 cpu_manager.go:414] "containerMap: removing container" podUID="bf81ff7f-b3b2-4821-abb5-258ed0bea849" containerName="revoke-keys"
Nov 02 09:00:00.138160 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.138128    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf81ff7f-b3b2-4821-abb5-258ed0bea849" containerName="revoke-keys"
Nov 02 09:00:00.138242 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.138212    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf81ff7f-b3b2-4821-abb5-258ed0bea849" containerName="revoke-keys"
Nov 02 09:00:00.141387 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.141361    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"
Nov 02 09:00:00.161095 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.161065    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"]
Nov 02 09:00:00.161422 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.161399    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893500-szqnb\" (UID: \"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"
Nov 02 09:00:00.161510 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.161435    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9l8\" (UniqueName: \"kubernetes.io/projected/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-kube-api-access-hf9l8\") pod \"maas-api-key-cleanup-29893500-szqnb\" (UID: \"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"
Nov 02 09:00:00.168939 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.168902    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:00:00.168939 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.168927    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:00:00.169140 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.169034    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:00:00.262347 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.262312    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893500-szqnb\" (UID: \"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"
Nov 02 09:00:00.262347 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.262346    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9l8\" (UniqueName: \"kubernetes.io/projected/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-kube-api-access-hf9l8\") pod \"maas-api-key-cleanup-29893500-szqnb\" (UID: \"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"
Nov 02 09:00:00.262978 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.262956    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893500-szqnb\" (UID: \"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"
Nov 02 09:00:00.286078 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.286053    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9l8\" (UniqueName: \"kubernetes.io/projected/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-kube-api-access-hf9l8\") pod \"maas-api-key-cleanup-29893500-szqnb\" (UID: \"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"
Nov 02 09:00:00.454658 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.454628    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"
Nov 02 09:00:00.755916 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:00.755862    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"]
Nov 02 09:00:00.757329 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:00:00.757300    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc6cc76_3753_4bc6_9fd8_cbe09e2af8be.slice/crio-87445aae9996e412734c8a1f3b00d1779375ee4eff2c78845a26b6d5f5c70369 WatchSource:0}: Error finding container 87445aae9996e412734c8a1f3b00d1779375ee4eff2c78845a26b6d5f5c70369: Status 404 returned error can't find the container with id 87445aae9996e412734c8a1f3b00d1779375ee4eff2c78845a26b6d5f5c70369
Nov 02 09:00:01.322696 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:01.322610    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb" event={"ID":"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be","Type":"ContainerStarted","Data":"7555e14a88b59cf15489e176c7b605f143383f1b28d2e58bacc8f943b96da86b"}
Nov 02 09:00:01.322696 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:01.322669    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb" event={"ID":"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be","Type":"ContainerStarted","Data":"87445aae9996e412734c8a1f3b00d1779375ee4eff2c78845a26b6d5f5c70369"}
Nov 02 09:00:01.364642 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:01.364590    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb" podStartSLOduration=1.364572954 podStartE2EDuration="1.364572954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:00:01.363434366 +0000 UTC m=+5752.053761331" watchObservedRunningTime="2026-11-02 09:00:01.364572954 +0000 UTC m=+5752.054899918"
Nov 02 09:00:02.329367 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:02.329331    4898 generic.go:356] "Generic (PLEG): container finished" podID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" containerID="7555e14a88b59cf15489e176c7b605f143383f1b28d2e58bacc8f943b96da86b" exitCode=60
Nov 02 09:00:02.329794 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:02.329386    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb" event={"ID":"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be","Type":"ContainerDied","Data":"7555e14a88b59cf15489e176c7b605f143383f1b28d2e58bacc8f943b96da86b"}
Nov 02 09:00:02.329794 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:02.329645    4898 scope.go:122] "RemoveContainer" containerID="7555e14a88b59cf15489e176c7b605f143383f1b28d2e58bacc8f943b96da86b"
Nov 02 09:00:03.337009 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:03.336925    4898 generic.go:356] "Generic (PLEG): container finished" podID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" containerID="ab0803fc2d89d94983af8f6c2a029c9641d7441dc014e67b3b8d6e3545512a98" exitCode=60
Nov 02 09:00:03.337405 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:03.337008    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb" event={"ID":"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be","Type":"ContainerDied","Data":"ab0803fc2d89d94983af8f6c2a029c9641d7441dc014e67b3b8d6e3545512a98"}
Nov 02 09:00:03.337405 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:03.337070    4898 scope.go:122] "RemoveContainer" containerID="7555e14a88b59cf15489e176c7b605f143383f1b28d2e58bacc8f943b96da86b"
Nov 02 09:00:03.337405 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:03.337364    4898 scope.go:122] "RemoveContainer" containerID="ab0803fc2d89d94983af8f6c2a029c9641d7441dc014e67b3b8d6e3545512a98"
Nov 02 09:00:03.337612 ip-10-0-134-79 kubenswrapper[4898]: E1102 09:00:03.337593    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893500-szqnb_odh-ai-gateway-infra(fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be"
Nov 02 09:00:04.344758 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:04.344729    4898 scope.go:122] "RemoveContainer" containerID="ab0803fc2d89d94983af8f6c2a029c9641d7441dc014e67b3b8d6e3545512a98"
Nov 02 09:00:04.345173 ip-10-0-134-79 kubenswrapper[4898]: E1102 09:00:04.344977    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893500-szqnb_odh-ai-gateway-infra(fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be"
Nov 02 09:00:14.989504 ip-10-0-134-79 kubenswrapper[4898]: E1102 09:00:14.989461    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 09:00:16.457269 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:16.457237    4898 scope.go:122] "RemoveContainer" containerID="ab0803fc2d89d94983af8f6c2a029c9641d7441dc014e67b3b8d6e3545512a98"
Nov 02 09:00:17.454199 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:17.454121    4898 generic.go:356] "Generic (PLEG): container finished" podID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" containerID="06e939862968c402aa76d9977b262128442dd4956f736fe4f27a8f4446bec53a" exitCode=60
Nov 02 09:00:17.454199 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:17.454154    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb" event={"ID":"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be","Type":"ContainerDied","Data":"06e939862968c402aa76d9977b262128442dd4956f736fe4f27a8f4446bec53a"}
Nov 02 09:00:17.454402 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:17.454232    4898 scope.go:122] "RemoveContainer" containerID="ab0803fc2d89d94983af8f6c2a029c9641d7441dc014e67b3b8d6e3545512a98"
Nov 02 09:00:17.454698 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:17.454676    4898 scope.go:122] "RemoveContainer" containerID="06e939862968c402aa76d9977b262128442dd4956f736fe4f27a8f4446bec53a"
Nov 02 09:00:17.455136 ip-10-0-134-79 kubenswrapper[4898]: E1102 09:00:17.455104    4898 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893500-szqnb_odh-ai-gateway-infra(fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be"
Nov 02 09:00:18.517027 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:18.516990    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"]
Nov 02 09:00:18.665631 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:18.665607    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"
Nov 02 09:00:18.775467 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:18.775370    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-kube-api-access-hf9l8\" (UniqueName: \"kubernetes.io/projected/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-kube-api-access-hf9l8\") pod \"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be\" (UID: \"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be\") "
Nov 02 09:00:18.775467 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:18.775425    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-maas-api-service-ca\") pod \"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be\" (UID: \"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be\") "
Nov 02 09:00:18.775879 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:18.775841    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-maas-api-service-ca" pod "fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" (UID: "fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:00:18.777740 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:18.777716    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-kube-api-access-hf9l8" pod "fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" (UID: "fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be"). InnerVolumeSpecName "kube-api-access-hf9l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:00:18.877307 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:18.877258    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf9l8\" (UniqueName: \"kubernetes.io/projected/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-kube-api-access-hf9l8\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:00:18.877307 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:18.877302    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be-maas-api-service-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:00:19.469967 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:19.469930    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb" event={"ID":"fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be","Type":"ContainerDied","Data":"87445aae9996e412734c8a1f3b00d1779375ee4eff2c78845a26b6d5f5c70369"}
Nov 02 09:00:19.470189 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:19.469975    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"
Nov 02 09:00:19.470189 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:19.469988    4898 scope.go:122] "RemoveContainer" containerID="06e939862968c402aa76d9977b262128442dd4956f736fe4f27a8f4446bec53a"
Nov 02 09:00:19.526112 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:19.526076    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"]
Nov 02 09:00:19.538196 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:19.538165    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893500-szqnb"]
Nov 02 09:00:21.464284 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:21.464247    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" path="/var/lib/kubelet/pods/fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be/volumes"
Nov 02 09:00:31.500689 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.500653    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"]
Nov 02 09:00:31.501317 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.501293    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" containerName="cleanup"
Nov 02 09:00:31.501416 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.501318    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" containerName="cleanup"
Nov 02 09:00:31.501416 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.501339    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" containerName="cleanup"
Nov 02 09:00:31.501416 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.501347    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" containerName="cleanup"
Nov 02 09:00:31.501580 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.501500    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" containerName="cleanup"
Nov 02 09:00:31.501580 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.501520    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" containerName="cleanup"
Nov 02 09:00:31.504957 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.504932    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:31.522834 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.522809    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:00:31.535049 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.535020    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:00:31.535253 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.535020    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:00:31.535253 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.535104    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:00:31.537935 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.537911    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"]
Nov 02 09:00:31.549423 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.549390    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8c45bb-g87bx\" (UID: \"a2fcbcba-e847-42e3-a9d7-680aac367b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:31.549423 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.549433    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f6gv\" (UniqueName: \"kubernetes.io/projected/a2fcbcba-e847-42e3-a9d7-680aac367b00-kube-api-access-2f6gv\") pod \"maas-api-revoke-sub-ce8c45bb-g87bx\" (UID: \"a2fcbcba-e847-42e3-a9d7-680aac367b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:31.549645 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.549576    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8c45bb-g87bx\" (UID: \"a2fcbcba-e847-42e3-a9d7-680aac367b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:31.651067 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.651026    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8c45bb-g87bx\" (UID: \"a2fcbcba-e847-42e3-a9d7-680aac367b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:31.651067 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.651066    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f6gv\" (UniqueName: \"kubernetes.io/projected/a2fcbcba-e847-42e3-a9d7-680aac367b00-kube-api-access-2f6gv\") pod \"maas-api-revoke-sub-ce8c45bb-g87bx\" (UID: \"a2fcbcba-e847-42e3-a9d7-680aac367b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:31.651358 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.651115    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8c45bb-g87bx\" (UID: \"a2fcbcba-e847-42e3-a9d7-680aac367b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:31.651746 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.651722    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8c45bb-g87bx\" (UID: \"a2fcbcba-e847-42e3-a9d7-680aac367b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:31.653541 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.653516    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8c45bb-g87bx\" (UID: \"a2fcbcba-e847-42e3-a9d7-680aac367b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:31.679569 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.679539    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f6gv\" (UniqueName: \"kubernetes.io/projected/a2fcbcba-e847-42e3-a9d7-680aac367b00-kube-api-access-2f6gv\") pod \"maas-api-revoke-sub-ce8c45bb-g87bx\" (UID: \"a2fcbcba-e847-42e3-a9d7-680aac367b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:31.819252 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:31.819162    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:32.122533 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:32.122499    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"]
Nov 02 09:00:32.124047 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:00:32.124017    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fcbcba_e847_42e3_a9d7_680aac367b00.slice/crio-497a7a66f48e4591dbb22765ad04f8ecbce1d8b72553c5773589645c3c8da8dd WatchSource:0}: Error finding container 497a7a66f48e4591dbb22765ad04f8ecbce1d8b72553c5773589645c3c8da8dd: Status 404 returned error can't find the container with id 497a7a66f48e4591dbb22765ad04f8ecbce1d8b72553c5773589645c3c8da8dd
Nov 02 09:00:32.598822 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:32.598782    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx" event={"ID":"a2fcbcba-e847-42e3-a9d7-680aac367b00","Type":"ContainerStarted","Data":"d7f1081f176798a464af4c9f8f1cd3dd105c0f65aa77098a6dc02e5e09ae3c62"}
Nov 02 09:00:32.599270 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:32.598844    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx" event={"ID":"a2fcbcba-e847-42e3-a9d7-680aac367b00","Type":"ContainerStarted","Data":"497a7a66f48e4591dbb22765ad04f8ecbce1d8b72553c5773589645c3c8da8dd"}
Nov 02 09:00:32.640375 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:32.640319    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx" podStartSLOduration=1.6403063009999999 podStartE2EDuration="1.640306301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:00:32.639464588 +0000 UTC m=+5783.329791553" watchObservedRunningTime="2026-11-02 09:00:32.640306301 +0000 UTC m=+5783.330633264"
Nov 02 09:00:33.606317 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:33.606276    4898 generic.go:356] "Generic (PLEG): container finished" podID="a2fcbcba-e847-42e3-a9d7-680aac367b00" containerID="d7f1081f176798a464af4c9f8f1cd3dd105c0f65aa77098a6dc02e5e09ae3c62" exitCode=0
Nov 02 09:00:33.606765 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:33.606360    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx" event={"ID":"a2fcbcba-e847-42e3-a9d7-680aac367b00","Type":"ContainerDied","Data":"d7f1081f176798a464af4c9f8f1cd3dd105c0f65aa77098a6dc02e5e09ae3c62"}
Nov 02 09:00:34.751609 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:34.751578    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:34.788743 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:34.788712    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fcbcba-e847-42e3-a9d7-680aac367b00-kube-api-access-2f6gv\" (UniqueName: \"kubernetes.io/projected/a2fcbcba-e847-42e3-a9d7-680aac367b00-kube-api-access-2f6gv\") pod \"a2fcbcba-e847-42e3-a9d7-680aac367b00\" (UID: \"a2fcbcba-e847-42e3-a9d7-680aac367b00\") "
Nov 02 09:00:34.788942 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:34.788799    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-xks-ca\") pod \"a2fcbcba-e847-42e3-a9d7-680aac367b00\" (UID: \"a2fcbcba-e847-42e3-a9d7-680aac367b00\") "
Nov 02 09:00:34.789005 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:34.788937    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-openshift-ca\") pod \"a2fcbcba-e847-42e3-a9d7-680aac367b00\" (UID: \"a2fcbcba-e847-42e3-a9d7-680aac367b00\") "
Nov 02 09:00:34.789301 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:34.789276    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-openshift-ca" pod "a2fcbcba-e847-42e3-a9d7-680aac367b00" (UID: "a2fcbcba-e847-42e3-a9d7-680aac367b00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:00:34.791296 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:34.791267    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-xks-ca" pod "a2fcbcba-e847-42e3-a9d7-680aac367b00" (UID: "a2fcbcba-e847-42e3-a9d7-680aac367b00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:00:34.791455 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:34.791403    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fcbcba-e847-42e3-a9d7-680aac367b00-kube-api-access-2f6gv" pod "a2fcbcba-e847-42e3-a9d7-680aac367b00" (UID: "a2fcbcba-e847-42e3-a9d7-680aac367b00"). InnerVolumeSpecName "kube-api-access-2f6gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:00:34.890457 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:34.890355    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f6gv\" (UniqueName: \"kubernetes.io/projected/a2fcbcba-e847-42e3-a9d7-680aac367b00-kube-api-access-2f6gv\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:00:34.890457 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:34.890405    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:00:34.890457 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:34.890416    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2fcbcba-e847-42e3-a9d7-680aac367b00-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:00:35.620349 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:35.620309    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx" event={"ID":"a2fcbcba-e847-42e3-a9d7-680aac367b00","Type":"ContainerDied","Data":"497a7a66f48e4591dbb22765ad04f8ecbce1d8b72553c5773589645c3c8da8dd"}
Nov 02 09:00:35.620571 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:35.620350    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8c45bb-g87bx"
Nov 02 09:00:35.620571 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:35.620361    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="497a7a66f48e4591dbb22765ad04f8ecbce1d8b72553c5773589645c3c8da8dd"
Nov 02 09:00:42.019947 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.019904    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"]
Nov 02 09:00:42.020659 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.020633    4898 cpu_manager.go:414] "containerMap: removing container" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" containerName="cleanup"
Nov 02 09:00:42.020728 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.020664    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" containerName="cleanup"
Nov 02 09:00:42.020770 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.020730    4898 cpu_manager.go:414] "containerMap: removing container" podUID="a2fcbcba-e847-42e3-a9d7-680aac367b00" containerName="revoke-keys"
Nov 02 09:00:42.020770 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.020740    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fcbcba-e847-42e3-a9d7-680aac367b00" containerName="revoke-keys"
Nov 02 09:00:42.020878 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.020863    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fcbcba-e847-42e3-a9d7-680aac367b00" containerName="revoke-keys"
Nov 02 09:00:42.020942 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.020901    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc6cc76-3753-4bc6-9fd8-cbe09e2af8be" containerName="cleanup"
Nov 02 09:00:42.024791 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.024772    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:42.042308 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.042272    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:00:42.042460 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.042313    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:00:42.054583 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.054558    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:00:42.054583 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.054557    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:00:42.066771 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.066738    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"]
Nov 02 09:00:42.163512 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.163473    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-759d1499-djg8j\" (UID: \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:42.163512 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.163516    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk6zk\" (UniqueName: \"kubernetes.io/projected/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-kube-api-access-nk6zk\") pod \"maas-api-revoke-sub-759d1499-djg8j\" (UID: \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:42.163732 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.163665    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-759d1499-djg8j\" (UID: \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:42.264681 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.264631    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-759d1499-djg8j\" (UID: \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:42.264878 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.264730    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-759d1499-djg8j\" (UID: \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:42.264878 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.264759    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk6zk\" (UniqueName: \"kubernetes.io/projected/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-kube-api-access-nk6zk\") pod \"maas-api-revoke-sub-759d1499-djg8j\" (UID: \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:42.265560 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.265529    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-759d1499-djg8j\" (UID: \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:42.267198 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.267174    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-759d1499-djg8j\" (UID: \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:42.289564 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.289490    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk6zk\" (UniqueName: \"kubernetes.io/projected/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-kube-api-access-nk6zk\") pod \"maas-api-revoke-sub-759d1499-djg8j\" (UID: \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:42.338397 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.338357    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:42.649414 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.649382    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"]
Nov 02 09:00:42.651151 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:00:42.651125    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78457a2_98e2_4db6_a4a1_97fcc4c9fd24.slice/crio-2362a80f1154946cce34a7759f5848aa25f63046952a9c5a56ac633d5d1fb152 WatchSource:0}: Error finding container 2362a80f1154946cce34a7759f5848aa25f63046952a9c5a56ac633d5d1fb152: Status 404 returned error can't find the container with id 2362a80f1154946cce34a7759f5848aa25f63046952a9c5a56ac633d5d1fb152
Nov 02 09:00:42.662308 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:42.662279    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j" event={"ID":"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24","Type":"ContainerStarted","Data":"2362a80f1154946cce34a7759f5848aa25f63046952a9c5a56ac633d5d1fb152"}
Nov 02 09:00:43.672968 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:43.672921    4898 generic.go:356] "Generic (PLEG): container finished" podID="e78457a2-98e2-4db6-a4a1-97fcc4c9fd24" containerID="259277f5ea929398d21cf4ddf6756414dcf0eefd69d9fe7e709e23f849683bdc" exitCode=0
Nov 02 09:00:43.673474 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:43.673058    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j" event={"ID":"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24","Type":"ContainerDied","Data":"259277f5ea929398d21cf4ddf6756414dcf0eefd69d9fe7e709e23f849683bdc"}
Nov 02 09:00:44.871973 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:44.871938    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:44.991114 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:44.991017    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-xks-ca\") pod \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\" (UID: \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\") "
Nov 02 09:00:44.991349 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:44.991127    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-openshift-ca\") pod \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\" (UID: \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\") "
Nov 02 09:00:44.991349 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:44.991259    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-kube-api-access-nk6zk\" (UniqueName: \"kubernetes.io/projected/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-kube-api-access-nk6zk\") pod \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\" (UID: \"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24\") "
Nov 02 09:00:44.992330 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:44.992298    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-openshift-ca" pod "e78457a2-98e2-4db6-a4a1-97fcc4c9fd24" (UID: "e78457a2-98e2-4db6-a4a1-97fcc4c9fd24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:00:44.994172 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:44.994140    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-kube-api-access-nk6zk" pod "e78457a2-98e2-4db6-a4a1-97fcc4c9fd24" (UID: "e78457a2-98e2-4db6-a4a1-97fcc4c9fd24"). InnerVolumeSpecName "kube-api-access-nk6zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:00:44.994780 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:44.994742    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-xks-ca" pod "e78457a2-98e2-4db6-a4a1-97fcc4c9fd24" (UID: "e78457a2-98e2-4db6-a4a1-97fcc4c9fd24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:00:45.001868 ip-10-0-134-79 kubenswrapper[4898]: E1102 09:00:45.001777    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 09:00:45.093179 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:45.093132    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk6zk\" (UniqueName: \"kubernetes.io/projected/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-kube-api-access-nk6zk\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:00:45.093179 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:45.093177    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:00:45.093473 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:45.093195    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78457a2-98e2-4db6-a4a1-97fcc4c9fd24-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:00:45.711847 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:45.711807    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j" event={"ID":"e78457a2-98e2-4db6-a4a1-97fcc4c9fd24","Type":"ContainerDied","Data":"2362a80f1154946cce34a7759f5848aa25f63046952a9c5a56ac633d5d1fb152"}
Nov 02 09:00:45.712180 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:45.712155    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2362a80f1154946cce34a7759f5848aa25f63046952a9c5a56ac633d5d1fb152"
Nov 02 09:00:45.723041 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:45.722066    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-759d1499-djg8j"
Nov 02 09:00:52.321181 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.321074    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"]
Nov 02 09:00:52.321789 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.321522    4898 cpu_manager.go:414] "containerMap: removing container" podUID="e78457a2-98e2-4db6-a4a1-97fcc4c9fd24" containerName="revoke-keys"
Nov 02 09:00:52.321789 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.321533    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78457a2-98e2-4db6-a4a1-97fcc4c9fd24" containerName="revoke-keys"
Nov 02 09:00:52.321789 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.321607    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78457a2-98e2-4db6-a4a1-97fcc4c9fd24" containerName="revoke-keys"
Nov 02 09:00:52.325224 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.325192    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:52.344344 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.344311    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"]
Nov 02 09:00:52.352518 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.352492    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:00:52.352715 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.352526    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:00:52.352715 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.352556    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:00:52.353261 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.353239    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:00:52.398733 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.398700    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f46cee4c-jx54p\" (UID: \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:52.398934 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.398768    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f46cee4c-jx54p\" (UID: \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:52.398934 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.398921    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jksn\" (UniqueName: \"kubernetes.io/projected/1fa1ee89-5eb3-4c8e-8e48-518750220e77-kube-api-access-9jksn\") pod \"maas-api-revoke-sub-f46cee4c-jx54p\" (UID: \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:52.499773 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.499735    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f46cee4c-jx54p\" (UID: \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:52.500031 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.499794    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jksn\" (UniqueName: \"kubernetes.io/projected/1fa1ee89-5eb3-4c8e-8e48-518750220e77-kube-api-access-9jksn\") pod \"maas-api-revoke-sub-f46cee4c-jx54p\" (UID: \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:52.500031 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.499868    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f46cee4c-jx54p\" (UID: \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:52.500536 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.500508    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f46cee4c-jx54p\" (UID: \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:52.502370 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.502341    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f46cee4c-jx54p\" (UID: \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:52.524003 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.523974    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jksn\" (UniqueName: \"kubernetes.io/projected/1fa1ee89-5eb3-4c8e-8e48-518750220e77-kube-api-access-9jksn\") pod \"maas-api-revoke-sub-f46cee4c-jx54p\" (UID: \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:52.638949 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.638836    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:52.939483 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:52.939453    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"]
Nov 02 09:00:52.940604 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:00:52.940574    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa1ee89_5eb3_4c8e_8e48_518750220e77.slice/crio-a9b1ce8159d261c4933d10fb99fd955b70c4ebedf3c1b6d0185cbc55287d19c2 WatchSource:0}: Error finding container a9b1ce8159d261c4933d10fb99fd955b70c4ebedf3c1b6d0185cbc55287d19c2: Status 404 returned error can't find the container with id a9b1ce8159d261c4933d10fb99fd955b70c4ebedf3c1b6d0185cbc55287d19c2
Nov 02 09:00:53.781498 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:53.781462    4898 generic.go:356] "Generic (PLEG): container finished" podID="1fa1ee89-5eb3-4c8e-8e48-518750220e77" containerID="edc0414ae54fcd1d96777eb95ca22d9d768eea2ca16e779779ea7c0864b79886" exitCode=0
Nov 02 09:00:53.781919 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:53.781549    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p" event={"ID":"1fa1ee89-5eb3-4c8e-8e48-518750220e77","Type":"ContainerDied","Data":"edc0414ae54fcd1d96777eb95ca22d9d768eea2ca16e779779ea7c0864b79886"}
Nov 02 09:00:53.781919 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:53.781606    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p" event={"ID":"1fa1ee89-5eb3-4c8e-8e48-518750220e77","Type":"ContainerStarted","Data":"a9b1ce8159d261c4933d10fb99fd955b70c4ebedf3c1b6d0185cbc55287d19c2"}
Nov 02 09:00:54.931216 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:54.931187    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:55.022506 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:55.022476    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa1ee89-5eb3-4c8e-8e48-518750220e77-kube-api-access-9jksn\" (UniqueName: \"kubernetes.io/projected/1fa1ee89-5eb3-4c8e-8e48-518750220e77-kube-api-access-9jksn\") pod \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\" (UID: \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\") "
Nov 02 09:00:55.022672 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:55.022519    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-xks-ca\") pod \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\" (UID: \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\") "
Nov 02 09:00:55.022672 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:55.022600    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-openshift-ca\") pod \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\" (UID: \"1fa1ee89-5eb3-4c8e-8e48-518750220e77\") "
Nov 02 09:00:55.023041 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:55.023012    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-openshift-ca" pod "1fa1ee89-5eb3-4c8e-8e48-518750220e77" (UID: "1fa1ee89-5eb3-4c8e-8e48-518750220e77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:00:55.024594 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:55.024571    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa1ee89-5eb3-4c8e-8e48-518750220e77-kube-api-access-9jksn" pod "1fa1ee89-5eb3-4c8e-8e48-518750220e77" (UID: "1fa1ee89-5eb3-4c8e-8e48-518750220e77"). InnerVolumeSpecName "kube-api-access-9jksn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:00:55.024662 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:55.024634    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-xks-ca" pod "1fa1ee89-5eb3-4c8e-8e48-518750220e77" (UID: "1fa1ee89-5eb3-4c8e-8e48-518750220e77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:00:55.124384 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:55.124302    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:00:55.124384 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:55.124331    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jksn\" (UniqueName: \"kubernetes.io/projected/1fa1ee89-5eb3-4c8e-8e48-518750220e77-kube-api-access-9jksn\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:00:55.124384 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:55.124341    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa1ee89-5eb3-4c8e-8e48-518750220e77-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:00:55.796711 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:55.796624    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p"
Nov 02 09:00:55.796882 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:55.796619    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f46cee4c-jx54p" event={"ID":"1fa1ee89-5eb3-4c8e-8e48-518750220e77","Type":"ContainerDied","Data":"a9b1ce8159d261c4933d10fb99fd955b70c4ebedf3c1b6d0185cbc55287d19c2"}
Nov 02 09:00:55.796882 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:00:55.796760    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b1ce8159d261c4933d10fb99fd955b70c4ebedf3c1b6d0185cbc55287d19c2"
Nov 02 09:01:02.791575 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.791520    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"]
Nov 02 09:01:02.792264 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.792241    4898 cpu_manager.go:414] "containerMap: removing container" podUID="1fa1ee89-5eb3-4c8e-8e48-518750220e77" containerName="revoke-keys"
Nov 02 09:01:02.792377 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.792266    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa1ee89-5eb3-4c8e-8e48-518750220e77" containerName="revoke-keys"
Nov 02 09:01:02.792443 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.792415    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa1ee89-5eb3-4c8e-8e48-518750220e77" containerName="revoke-keys"
Nov 02 09:01:02.795772 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.795748    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:02.817078 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.817045    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"]
Nov 02 09:01:02.824377 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.824348    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:01:02.824535 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.824442    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:01:02.839939 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.839898    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:01:02.840192 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.840166    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:01:02.900931 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.900876    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbtv\" (UniqueName: \"kubernetes.io/projected/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-kube-api-access-qvbtv\") pod \"maas-api-revoke-sub-ed246961-b59j7\" (UID: \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:02.901232 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.901027    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed246961-b59j7\" (UID: \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:02.901232 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:02.901082    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed246961-b59j7\" (UID: \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:03.002028 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:03.001989    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed246961-b59j7\" (UID: \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:03.002185 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:03.002046    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed246961-b59j7\" (UID: \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:03.002185 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:03.002127    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbtv\" (UniqueName: \"kubernetes.io/projected/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-kube-api-access-qvbtv\") pod \"maas-api-revoke-sub-ed246961-b59j7\" (UID: \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:03.002829 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:03.002801    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed246961-b59j7\" (UID: \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:03.004580 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:03.004556    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed246961-b59j7\" (UID: \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:03.049641 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:03.049564    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbtv\" (UniqueName: \"kubernetes.io/projected/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-kube-api-access-qvbtv\") pod \"maas-api-revoke-sub-ed246961-b59j7\" (UID: \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:03.109620 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:03.109576    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:03.417338 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:03.417307    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"]
Nov 02 09:01:03.418320 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:01:03.418289    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c30fd2_94de_42a0_88bf_5fc275e1fc34.slice/crio-b39c730f7dff2ff69c1f559a5497cf94f80c0cba5f427f4404533bd34aa53bfc WatchSource:0}: Error finding container b39c730f7dff2ff69c1f559a5497cf94f80c0cba5f427f4404533bd34aa53bfc: Status 404 returned error can't find the container with id b39c730f7dff2ff69c1f559a5497cf94f80c0cba5f427f4404533bd34aa53bfc
Nov 02 09:01:03.846771 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:03.846734    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7" event={"ID":"d0c30fd2-94de-42a0-88bf-5fc275e1fc34","Type":"ContainerStarted","Data":"8c88b7ea8226ac12b30c51d08dc49779a45617945dbc659578085b8a437eb02a"}
Nov 02 09:01:03.847197 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:03.846801    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7" event={"ID":"d0c30fd2-94de-42a0-88bf-5fc275e1fc34","Type":"ContainerStarted","Data":"b39c730f7dff2ff69c1f559a5497cf94f80c0cba5f427f4404533bd34aa53bfc"}
Nov 02 09:01:03.891969 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:03.891912    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7" podStartSLOduration=1.891869277 podStartE2EDuration="1.891869277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:01:03.890809341 +0000 UTC m=+5814.581136310" watchObservedRunningTime="2026-11-02 09:01:03.891869277 +0000 UTC m=+5814.582196241"
Nov 02 09:01:04.858601 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:04.858560    4898 generic.go:356] "Generic (PLEG): container finished" podID="d0c30fd2-94de-42a0-88bf-5fc275e1fc34" containerID="8c88b7ea8226ac12b30c51d08dc49779a45617945dbc659578085b8a437eb02a" exitCode=0
Nov 02 09:01:04.859222 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:04.858658    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7" event={"ID":"d0c30fd2-94de-42a0-88bf-5fc275e1fc34","Type":"ContainerDied","Data":"8c88b7ea8226ac12b30c51d08dc49779a45617945dbc659578085b8a437eb02a"}
Nov 02 09:01:06.419360 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.419322    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:06.449577 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.449540    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-xks-ca\") pod \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\" (UID: \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\") "
Nov 02 09:01:06.449743 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.449594    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-kube-api-access-qvbtv\" (UniqueName: \"kubernetes.io/projected/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-kube-api-access-qvbtv\") pod \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\" (UID: \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\") "
Nov 02 09:01:06.449743 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.449710    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-openshift-ca\") pod \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\" (UID: \"d0c30fd2-94de-42a0-88bf-5fc275e1fc34\") "
Nov 02 09:01:06.450124 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.450097    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-openshift-ca" pod "d0c30fd2-94de-42a0-88bf-5fc275e1fc34" (UID: "d0c30fd2-94de-42a0-88bf-5fc275e1fc34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:01:06.450230 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.450210    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:06.451657 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.451631    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-xks-ca" pod "d0c30fd2-94de-42a0-88bf-5fc275e1fc34" (UID: "d0c30fd2-94de-42a0-88bf-5fc275e1fc34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:01:06.451746 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.451700    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-kube-api-access-qvbtv" pod "d0c30fd2-94de-42a0-88bf-5fc275e1fc34" (UID: "d0c30fd2-94de-42a0-88bf-5fc275e1fc34"). InnerVolumeSpecName "kube-api-access-qvbtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:01:06.551858 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.551804    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:06.551858 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.551857    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbtv\" (UniqueName: \"kubernetes.io/projected/d0c30fd2-94de-42a0-88bf-5fc275e1fc34-kube-api-access-qvbtv\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:06.879777 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.879733    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7" event={"ID":"d0c30fd2-94de-42a0-88bf-5fc275e1fc34","Type":"ContainerDied","Data":"b39c730f7dff2ff69c1f559a5497cf94f80c0cba5f427f4404533bd34aa53bfc"}
Nov 02 09:01:06.880188 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.880155    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed246961-b59j7"
Nov 02 09:01:06.880368 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:06.880347    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b39c730f7dff2ff69c1f559a5497cf94f80c0cba5f427f4404533bd34aa53bfc"
Nov 02 09:01:13.259952 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.259915    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"]
Nov 02 09:01:13.260429 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.260414    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d0c30fd2-94de-42a0-88bf-5fc275e1fc34" containerName="revoke-keys"
Nov 02 09:01:13.260482 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.260431    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c30fd2-94de-42a0-88bf-5fc275e1fc34" containerName="revoke-keys"
Nov 02 09:01:13.260560 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.260550    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c30fd2-94de-42a0-88bf-5fc275e1fc34" containerName="revoke-keys"
Nov 02 09:01:13.264290 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.264263    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:13.295712 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.295677    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:01:13.296020 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.295975    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:01:13.296020 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.295844    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:01:13.296020 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.295757    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:01:13.299249 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.299220    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"]
Nov 02 09:01:13.319078 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.319030    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7tg9\" (UniqueName: \"kubernetes.io/projected/ad797edc-37b9-4558-82ce-c23c46a5def3-kube-api-access-b7tg9\") pod \"maas-api-revoke-sub-f95b0dd8-9skn4\" (UID: \"ad797edc-37b9-4558-82ce-c23c46a5def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:13.319292 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.319237    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f95b0dd8-9skn4\" (UID: \"ad797edc-37b9-4558-82ce-c23c46a5def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:13.319373 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.319317    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f95b0dd8-9skn4\" (UID: \"ad797edc-37b9-4558-82ce-c23c46a5def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:13.420918 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.420865    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f95b0dd8-9skn4\" (UID: \"ad797edc-37b9-4558-82ce-c23c46a5def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:13.421085 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.421012    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f95b0dd8-9skn4\" (UID: \"ad797edc-37b9-4558-82ce-c23c46a5def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:13.421085 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.421046    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7tg9\" (UniqueName: \"kubernetes.io/projected/ad797edc-37b9-4558-82ce-c23c46a5def3-kube-api-access-b7tg9\") pod \"maas-api-revoke-sub-f95b0dd8-9skn4\" (UID: \"ad797edc-37b9-4558-82ce-c23c46a5def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:13.421680 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.421650    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f95b0dd8-9skn4\" (UID: \"ad797edc-37b9-4558-82ce-c23c46a5def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:13.423571 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.423544    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f95b0dd8-9skn4\" (UID: \"ad797edc-37b9-4558-82ce-c23c46a5def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:13.446458 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.446429    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7tg9\" (UniqueName: \"kubernetes.io/projected/ad797edc-37b9-4558-82ce-c23c46a5def3-kube-api-access-b7tg9\") pod \"maas-api-revoke-sub-f95b0dd8-9skn4\" (UID: \"ad797edc-37b9-4558-82ce-c23c46a5def3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:13.580969 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.580840    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:13.891101 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.891068    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"]
Nov 02 09:01:13.891237 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:01:13.891070    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad797edc_37b9_4558_82ce_c23c46a5def3.slice/crio-590a28234e2dddac9a0201fa7365c6ae2f5eba7c07ea5a7168cda025a64949c9 WatchSource:0}: Error finding container 590a28234e2dddac9a0201fa7365c6ae2f5eba7c07ea5a7168cda025a64949c9: Status 404 returned error can't find the container with id 590a28234e2dddac9a0201fa7365c6ae2f5eba7c07ea5a7168cda025a64949c9
Nov 02 09:01:13.929428 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:13.929395    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4" event={"ID":"ad797edc-37b9-4558-82ce-c23c46a5def3","Type":"ContainerStarted","Data":"590a28234e2dddac9a0201fa7365c6ae2f5eba7c07ea5a7168cda025a64949c9"}
Nov 02 09:01:14.939731 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:14.939688    4898 generic.go:356] "Generic (PLEG): container finished" podID="ad797edc-37b9-4558-82ce-c23c46a5def3" containerID="56a9fcc49661ccb8eea955ca079bde8853a6f77fcb09fddaf1abf4148ae5dbc5" exitCode=0
Nov 02 09:01:14.940313 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:14.939929    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4" event={"ID":"ad797edc-37b9-4558-82ce-c23c46a5def3","Type":"ContainerDied","Data":"56a9fcc49661ccb8eea955ca079bde8853a6f77fcb09fddaf1abf4148ae5dbc5"}
Nov 02 09:01:15.001362 ip-10-0-134-79 kubenswrapper[4898]: E1102 09:01:15.001315    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 09:01:16.451178 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.451139    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:16.647298 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.647211    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-xks-ca\") pod \"ad797edc-37b9-4558-82ce-c23c46a5def3\" (UID: \"ad797edc-37b9-4558-82ce-c23c46a5def3\") "
Nov 02 09:01:16.647462 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.647322    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-openshift-ca\") pod \"ad797edc-37b9-4558-82ce-c23c46a5def3\" (UID: \"ad797edc-37b9-4558-82ce-c23c46a5def3\") "
Nov 02 09:01:16.647462 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.647404    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad797edc-37b9-4558-82ce-c23c46a5def3-kube-api-access-b7tg9\" (UniqueName: \"kubernetes.io/projected/ad797edc-37b9-4558-82ce-c23c46a5def3-kube-api-access-b7tg9\") pod \"ad797edc-37b9-4558-82ce-c23c46a5def3\" (UID: \"ad797edc-37b9-4558-82ce-c23c46a5def3\") "
Nov 02 09:01:16.648435 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.648411    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-openshift-ca" pod "ad797edc-37b9-4558-82ce-c23c46a5def3" (UID: "ad797edc-37b9-4558-82ce-c23c46a5def3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:01:16.649934 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.649900    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-xks-ca" pod "ad797edc-37b9-4558-82ce-c23c46a5def3" (UID: "ad797edc-37b9-4558-82ce-c23c46a5def3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:01:16.650274 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.650256    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad797edc-37b9-4558-82ce-c23c46a5def3-kube-api-access-b7tg9" pod "ad797edc-37b9-4558-82ce-c23c46a5def3" (UID: "ad797edc-37b9-4558-82ce-c23c46a5def3"). InnerVolumeSpecName "kube-api-access-b7tg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:01:16.748195 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.748152    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7tg9\" (UniqueName: \"kubernetes.io/projected/ad797edc-37b9-4558-82ce-c23c46a5def3-kube-api-access-b7tg9\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:16.748195 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.748182    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:16.748195 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.748194    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad797edc-37b9-4558-82ce-c23c46a5def3-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:17.000914 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.992263    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4" event={"ID":"ad797edc-37b9-4558-82ce-c23c46a5def3","Type":"ContainerDied","Data":"590a28234e2dddac9a0201fa7365c6ae2f5eba7c07ea5a7168cda025a64949c9"}
Nov 02 09:01:17.000914 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.992337    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="590a28234e2dddac9a0201fa7365c6ae2f5eba7c07ea5a7168cda025a64949c9"
Nov 02 09:01:17.000914 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:16.992421    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95b0dd8-9skn4"
Nov 02 09:01:23.730020 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.729982    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"]
Nov 02 09:01:23.730708 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.730684    4898 cpu_manager.go:414] "containerMap: removing container" podUID="ad797edc-37b9-4558-82ce-c23c46a5def3" containerName="revoke-keys"
Nov 02 09:01:23.730797 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.730710    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad797edc-37b9-4558-82ce-c23c46a5def3" containerName="revoke-keys"
Nov 02 09:01:23.730860 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.730823    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad797edc-37b9-4558-82ce-c23c46a5def3" containerName="revoke-keys"
Nov 02 09:01:23.734263 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.734235    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:23.752402 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.752371    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:01:23.764855 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.764820    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:01:23.765084 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.765043    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:01:23.765305 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.765281    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:01:23.768243 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.768217    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"]
Nov 02 09:01:23.804907 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.804865    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c713800e-xg5rh\" (UID: \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:23.805075 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.805018    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsphq\" (UniqueName: \"kubernetes.io/projected/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-kube-api-access-vsphq\") pod \"maas-api-revoke-sub-c713800e-xg5rh\" (UID: \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:23.805160 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.805142    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c713800e-xg5rh\" (UID: \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:23.906334 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.906297    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c713800e-xg5rh\" (UID: \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:23.906513 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.906347    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c713800e-xg5rh\" (UID: \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:23.906513 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.906406    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsphq\" (UniqueName: \"kubernetes.io/projected/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-kube-api-access-vsphq\") pod \"maas-api-revoke-sub-c713800e-xg5rh\" (UID: \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:23.907137 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.907110    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c713800e-xg5rh\" (UID: \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:23.908860 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.908829    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c713800e-xg5rh\" (UID: \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:23.929983 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:23.929953    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsphq\" (UniqueName: \"kubernetes.io/projected/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-kube-api-access-vsphq\") pod \"maas-api-revoke-sub-c713800e-xg5rh\" (UID: \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:24.047876 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:24.047790    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:24.360256 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:24.360159    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"]
Nov 02 09:01:24.365995 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:01:24.365959    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20df35b_d387_4a7e_b7ff_0a4cff8ecf4a.slice/crio-6c8f5a28f113610e13a7b378c40184650afb1ec993184762edf4e730b333adad WatchSource:0}: Error finding container 6c8f5a28f113610e13a7b378c40184650afb1ec993184762edf4e730b333adad: Status 404 returned error can't find the container with id 6c8f5a28f113610e13a7b378c40184650afb1ec993184762edf4e730b333adad
Nov 02 09:01:25.044351 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:25.044252    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh" event={"ID":"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a","Type":"ContainerStarted","Data":"55f3f328b33783918e42386ec3e8923360e127589bea855d775bf26b21b31bff"}
Nov 02 09:01:25.044842 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:25.044356    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh" event={"ID":"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a","Type":"ContainerStarted","Data":"6c8f5a28f113610e13a7b378c40184650afb1ec993184762edf4e730b333adad"}
Nov 02 09:01:25.097909 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:25.097312    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh" podStartSLOduration=2.097290724 podStartE2EDuration="2.097290724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:01:25.089676627 +0000 UTC m=+5835.780003597" watchObservedRunningTime="2026-11-02 09:01:25.097290724 +0000 UTC m=+5835.787617689"
Nov 02 09:01:26.065916 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:26.063783    4898 generic.go:356] "Generic (PLEG): container finished" podID="d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a" containerID="55f3f328b33783918e42386ec3e8923360e127589bea855d775bf26b21b31bff" exitCode=0
Nov 02 09:01:26.065916 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:26.063876    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh" event={"ID":"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a","Type":"ContainerDied","Data":"55f3f328b33783918e42386ec3e8923360e127589bea855d775bf26b21b31bff"}
Nov 02 09:01:27.543986 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:27.543948    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:27.550433 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:27.550411    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-xks-ca\") pod \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\" (UID: \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\") "
Nov 02 09:01:27.550535 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:27.550450    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-openshift-ca\") pod \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\" (UID: \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\") "
Nov 02 09:01:27.550535 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:27.550492    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-kube-api-access-vsphq\" (UniqueName: \"kubernetes.io/projected/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-kube-api-access-vsphq\") pod \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\" (UID: \"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a\") "
Nov 02 09:01:27.550841 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:27.550813    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-openshift-ca" pod "d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a" (UID: "d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:01:27.552654 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:27.552622    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-kube-api-access-vsphq" pod "d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a" (UID: "d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a"). InnerVolumeSpecName "kube-api-access-vsphq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:01:27.552793 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:27.552654    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-xks-ca" pod "d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a" (UID: "d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:01:27.651304 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:27.651263    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:27.651304 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:27.651300    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:27.651304 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:27.651314    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsphq\" (UniqueName: \"kubernetes.io/projected/d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a-kube-api-access-vsphq\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:28.095867 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:28.095831    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh" event={"ID":"d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a","Type":"ContainerDied","Data":"6c8f5a28f113610e13a7b378c40184650afb1ec993184762edf4e730b333adad"}
Nov 02 09:01:28.095867 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:28.095861    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c713800e-xg5rh"
Nov 02 09:01:28.096120 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:28.095884    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c8f5a28f113610e13a7b378c40184650afb1ec993184762edf4e730b333adad"
Nov 02 09:01:34.082855 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.082814    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"]
Nov 02 09:01:34.083397 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.083358    4898 cpu_manager.go:414] "containerMap: removing container" podUID="d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a" containerName="revoke-keys"
Nov 02 09:01:34.083397 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.083377    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a" containerName="revoke-keys"
Nov 02 09:01:34.083540 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.083463    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a" containerName="revoke-keys"
Nov 02 09:01:34.086682 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.086662    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:34.098813 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.098779    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62443a94-t8qh4\" (UID: \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:34.098967 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.098831    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glcqw\" (UniqueName: \"kubernetes.io/projected/57cabc7d-81a8-4408-9ed8-775abfb1e13e-kube-api-access-glcqw\") pod \"maas-api-revoke-sub-62443a94-t8qh4\" (UID: \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:34.098967 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.098918    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62443a94-t8qh4\" (UID: \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:34.114253 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.114216    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:01:34.114465 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.114265    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:01:34.114465 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.114355    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:01:34.114465 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.114372    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:01:34.117535 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.117499    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"]
Nov 02 09:01:34.199697 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.199653    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62443a94-t8qh4\" (UID: \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:34.199909 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.199728    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glcqw\" (UniqueName: \"kubernetes.io/projected/57cabc7d-81a8-4408-9ed8-775abfb1e13e-kube-api-access-glcqw\") pod \"maas-api-revoke-sub-62443a94-t8qh4\" (UID: \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:34.199909 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.199759    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62443a94-t8qh4\" (UID: \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:34.200511 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.200486    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62443a94-t8qh4\" (UID: \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:34.202215 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.202188    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62443a94-t8qh4\" (UID: \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:34.223534 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.223500    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glcqw\" (UniqueName: \"kubernetes.io/projected/57cabc7d-81a8-4408-9ed8-775abfb1e13e-kube-api-access-glcqw\") pod \"maas-api-revoke-sub-62443a94-t8qh4\" (UID: \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:34.406752 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.406719    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:34.710586 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:34.710546    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"]
Nov 02 09:01:34.712088 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:01:34.712058    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57cabc7d_81a8_4408_9ed8_775abfb1e13e.slice/crio-785b917adbc44b3ee57c60ce4c57c9eec6a6f6b077ef682dab24dd2ff085d0b5 WatchSource:0}: Error finding container 785b917adbc44b3ee57c60ce4c57c9eec6a6f6b077ef682dab24dd2ff085d0b5: Status 404 returned error can't find the container with id 785b917adbc44b3ee57c60ce4c57c9eec6a6f6b077ef682dab24dd2ff085d0b5
Nov 02 09:01:35.143508 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:35.143467    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4" event={"ID":"57cabc7d-81a8-4408-9ed8-775abfb1e13e","Type":"ContainerStarted","Data":"2e861b617641596b5701ad3f30cf0f9076a09f2f35936dab26d11232e342beab"}
Nov 02 09:01:35.143976 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:35.143534    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4" event={"ID":"57cabc7d-81a8-4408-9ed8-775abfb1e13e","Type":"ContainerStarted","Data":"785b917adbc44b3ee57c60ce4c57c9eec6a6f6b077ef682dab24dd2ff085d0b5"}
Nov 02 09:01:35.193969 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:35.193878    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4" podStartSLOduration=1.193857769 podStartE2EDuration="1.193857769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:01:35.193666595 +0000 UTC m=+5845.883993577" watchObservedRunningTime="2026-11-02 09:01:35.193857769 +0000 UTC m=+5845.884184735"
Nov 02 09:01:36.155328 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:36.155280    4898 generic.go:356] "Generic (PLEG): container finished" podID="57cabc7d-81a8-4408-9ed8-775abfb1e13e" containerID="2e861b617641596b5701ad3f30cf0f9076a09f2f35936dab26d11232e342beab" exitCode=0
Nov 02 09:01:36.155844 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:36.155346    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4" event={"ID":"57cabc7d-81a8-4408-9ed8-775abfb1e13e","Type":"ContainerDied","Data":"2e861b617641596b5701ad3f30cf0f9076a09f2f35936dab26d11232e342beab"}
Nov 02 09:01:37.634871 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:37.634831    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:37.751650 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:37.751566    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-openshift-ca\") pod \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\" (UID: \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\") "
Nov 02 09:01:37.751795 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:37.751650    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57cabc7d-81a8-4408-9ed8-775abfb1e13e-kube-api-access-glcqw\" (UniqueName: \"kubernetes.io/projected/57cabc7d-81a8-4408-9ed8-775abfb1e13e-kube-api-access-glcqw\") pod \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\" (UID: \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\") "
Nov 02 09:01:37.751795 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:37.751702    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-xks-ca\") pod \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\" (UID: \"57cabc7d-81a8-4408-9ed8-775abfb1e13e\") "
Nov 02 09:01:37.752280 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:37.752247    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-openshift-ca" pod "57cabc7d-81a8-4408-9ed8-775abfb1e13e" (UID: "57cabc7d-81a8-4408-9ed8-775abfb1e13e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:01:37.780054 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:37.777213    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-xks-ca" pod "57cabc7d-81a8-4408-9ed8-775abfb1e13e" (UID: "57cabc7d-81a8-4408-9ed8-775abfb1e13e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:01:37.805198 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:37.805154    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cabc7d-81a8-4408-9ed8-775abfb1e13e-kube-api-access-glcqw" pod "57cabc7d-81a8-4408-9ed8-775abfb1e13e" (UID: "57cabc7d-81a8-4408-9ed8-775abfb1e13e"). InnerVolumeSpecName "kube-api-access-glcqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:01:37.853210 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:37.853172    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:37.853210 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:37.853206    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glcqw\" (UniqueName: \"kubernetes.io/projected/57cabc7d-81a8-4408-9ed8-775abfb1e13e-kube-api-access-glcqw\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:37.853210 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:37.853216    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57cabc7d-81a8-4408-9ed8-775abfb1e13e-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:38.131224 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:38.131120    4898 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m4rk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:01:38.131541 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:38.131493    4898 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m4rk6" podUID="a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:01:38.186329 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:38.186279    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4" event={"ID":"57cabc7d-81a8-4408-9ed8-775abfb1e13e","Type":"ContainerDied","Data":"785b917adbc44b3ee57c60ce4c57c9eec6a6f6b077ef682dab24dd2ff085d0b5"}
Nov 02 09:01:38.186512 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:38.186353    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="785b917adbc44b3ee57c60ce4c57c9eec6a6f6b077ef682dab24dd2ff085d0b5"
Nov 02 09:01:38.186512 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:38.186430    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62443a94-t8qh4"
Nov 02 09:01:44.529959 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.529915    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"]
Nov 02 09:01:44.530620 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.530598    4898 cpu_manager.go:414] "containerMap: removing container" podUID="57cabc7d-81a8-4408-9ed8-775abfb1e13e" containerName="revoke-keys"
Nov 02 09:01:44.530697 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.530622    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57cabc7d-81a8-4408-9ed8-775abfb1e13e" containerName="revoke-keys"
Nov 02 09:01:44.530760 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.530747    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="57cabc7d-81a8-4408-9ed8-775abfb1e13e" containerName="revoke-keys"
Nov 02 09:01:44.534255 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.534233    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:44.557685 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.557645    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"]
Nov 02 09:01:44.566414 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.566381    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:01:44.566648 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.566440    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:01:44.575040 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.575004    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:01:44.575237 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.575101    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:01:44.633499 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.633459    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cqqp\" (UniqueName: \"kubernetes.io/projected/7b8144d3-2c7f-434a-8b0d-2bab4754388c-kube-api-access-5cqqp\") pod \"maas-api-revoke-sub-dd14d128-vmzxc\" (UID: \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:44.633499 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.633502    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd14d128-vmzxc\" (UID: \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:44.633741 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.633529    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd14d128-vmzxc\" (UID: \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:44.734637 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.734595    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cqqp\" (UniqueName: \"kubernetes.io/projected/7b8144d3-2c7f-434a-8b0d-2bab4754388c-kube-api-access-5cqqp\") pod \"maas-api-revoke-sub-dd14d128-vmzxc\" (UID: \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:44.734637 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.734639    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd14d128-vmzxc\" (UID: \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:44.734929 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.734656    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd14d128-vmzxc\" (UID: \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:44.735486 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.735451    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd14d128-vmzxc\" (UID: \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:44.737168 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.737147    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd14d128-vmzxc\" (UID: \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:44.758529 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.758492    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cqqp\" (UniqueName: \"kubernetes.io/projected/7b8144d3-2c7f-434a-8b0d-2bab4754388c-kube-api-access-5cqqp\") pod \"maas-api-revoke-sub-dd14d128-vmzxc\" (UID: \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:44.848119 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:44.848030    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:44.992537 ip-10-0-134-79 kubenswrapper[4898]: E1102 09:01:44.992495    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 09:01:45.161843 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:45.161814    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"]
Nov 02 09:01:45.162703 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:01:45.162677    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8144d3_2c7f_434a_8b0d_2bab4754388c.slice/crio-06ffb1c59f76c9e5e375b2714a62898c12362b4a8e109102e04a0303ba829d6e WatchSource:0}: Error finding container 06ffb1c59f76c9e5e375b2714a62898c12362b4a8e109102e04a0303ba829d6e: Status 404 returned error can't find the container with id 06ffb1c59f76c9e5e375b2714a62898c12362b4a8e109102e04a0303ba829d6e
Nov 02 09:01:45.229813 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:45.229771    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc" event={"ID":"7b8144d3-2c7f-434a-8b0d-2bab4754388c","Type":"ContainerStarted","Data":"06ffb1c59f76c9e5e375b2714a62898c12362b4a8e109102e04a0303ba829d6e"}
Nov 02 09:01:46.239178 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:46.239139    4898 generic.go:356] "Generic (PLEG): container finished" podID="7b8144d3-2c7f-434a-8b0d-2bab4754388c" containerID="87b8895efe20924e052442bb43e5bd4e702e70b88eaea07b09d6b38d0a5baf0b" exitCode=0
Nov 02 09:01:46.239610 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:46.239232    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc" event={"ID":"7b8144d3-2c7f-434a-8b0d-2bab4754388c","Type":"ContainerDied","Data":"87b8895efe20924e052442bb43e5bd4e702e70b88eaea07b09d6b38d0a5baf0b"}
Nov 02 09:01:47.846719 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:47.846678    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:47.945375 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:47.945337    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8144d3-2c7f-434a-8b0d-2bab4754388c-kube-api-access-5cqqp\" (UniqueName: \"kubernetes.io/projected/7b8144d3-2c7f-434a-8b0d-2bab4754388c-kube-api-access-5cqqp\") pod \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\" (UID: \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\") "
Nov 02 09:01:47.945564 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:47.945499    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-openshift-ca\") pod \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\" (UID: \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\") "
Nov 02 09:01:47.945564 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:47.945544    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-xks-ca\") pod \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\" (UID: \"7b8144d3-2c7f-434a-8b0d-2bab4754388c\") "
Nov 02 09:01:47.945994 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:47.945963    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-openshift-ca" pod "7b8144d3-2c7f-434a-8b0d-2bab4754388c" (UID: "7b8144d3-2c7f-434a-8b0d-2bab4754388c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:01:47.947621 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:47.947590    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-xks-ca" pod "7b8144d3-2c7f-434a-8b0d-2bab4754388c" (UID: "7b8144d3-2c7f-434a-8b0d-2bab4754388c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:01:47.947725 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:47.947637    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8144d3-2c7f-434a-8b0d-2bab4754388c-kube-api-access-5cqqp" pod "7b8144d3-2c7f-434a-8b0d-2bab4754388c" (UID: "7b8144d3-2c7f-434a-8b0d-2bab4754388c"). InnerVolumeSpecName "kube-api-access-5cqqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:01:48.047005 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:48.046872    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:48.047005 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:48.046940    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cqqp\" (UniqueName: \"kubernetes.io/projected/7b8144d3-2c7f-434a-8b0d-2bab4754388c-kube-api-access-5cqqp\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:48.047005 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:48.046957    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b8144d3-2c7f-434a-8b0d-2bab4754388c-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:48.272916 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:48.263763    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc" event={"ID":"7b8144d3-2c7f-434a-8b0d-2bab4754388c","Type":"ContainerDied","Data":"06ffb1c59f76c9e5e375b2714a62898c12362b4a8e109102e04a0303ba829d6e"}
Nov 02 09:01:48.272916 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:48.263837    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ffb1c59f76c9e5e375b2714a62898c12362b4a8e109102e04a0303ba829d6e"
Nov 02 09:01:48.272916 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:48.263943    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14d128-vmzxc"
Nov 02 09:01:54.877266 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:54.877227    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"]
Nov 02 09:01:54.877703 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:54.877684    4898 cpu_manager.go:414] "containerMap: removing container" podUID="7b8144d3-2c7f-434a-8b0d-2bab4754388c" containerName="revoke-keys"
Nov 02 09:01:54.877703 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:54.877696    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8144d3-2c7f-434a-8b0d-2bab4754388c" containerName="revoke-keys"
Nov 02 09:01:54.877780 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:54.877764    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8144d3-2c7f-434a-8b0d-2bab4754388c" containerName="revoke-keys"
Nov 02 09:01:54.881438 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:54.881416    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:54.900341 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:54.900310    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"]
Nov 02 09:01:54.907832 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:54.907804    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:01:54.908061 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:54.907911    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:01:54.948543 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:54.948507    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:01:54.948543 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:54.948524    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:01:55.013289 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:55.013250    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52820038-029c-4693-8905-7756b027c59a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2898fca-6thlq\" (UID: \"52820038-029c-4693-8905-7756b027c59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:55.013527 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:55.013335    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w8kv\" (UniqueName: \"kubernetes.io/projected/52820038-029c-4693-8905-7756b027c59a-kube-api-access-5w8kv\") pod \"maas-api-revoke-sub-a2898fca-6thlq\" (UID: \"52820038-029c-4693-8905-7756b027c59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:55.013527 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:55.013478    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52820038-029c-4693-8905-7756b027c59a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2898fca-6thlq\" (UID: \"52820038-029c-4693-8905-7756b027c59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:55.114192 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:55.114153    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52820038-029c-4693-8905-7756b027c59a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2898fca-6thlq\" (UID: \"52820038-029c-4693-8905-7756b027c59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:55.114418 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:55.114223    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52820038-029c-4693-8905-7756b027c59a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2898fca-6thlq\" (UID: \"52820038-029c-4693-8905-7756b027c59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:55.114418 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:55.114268    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w8kv\" (UniqueName: \"kubernetes.io/projected/52820038-029c-4693-8905-7756b027c59a-kube-api-access-5w8kv\") pod \"maas-api-revoke-sub-a2898fca-6thlq\" (UID: \"52820038-029c-4693-8905-7756b027c59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:55.114855 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:55.114828    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52820038-029c-4693-8905-7756b027c59a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2898fca-6thlq\" (UID: \"52820038-029c-4693-8905-7756b027c59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:55.116766 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:55.116741    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52820038-029c-4693-8905-7756b027c59a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2898fca-6thlq\" (UID: \"52820038-029c-4693-8905-7756b027c59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:55.138053 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:55.137971    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w8kv\" (UniqueName: \"kubernetes.io/projected/52820038-029c-4693-8905-7756b027c59a-kube-api-access-5w8kv\") pod \"maas-api-revoke-sub-a2898fca-6thlq\" (UID: \"52820038-029c-4693-8905-7756b027c59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:55.194693 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:55.194655    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:55.503733 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:01:55.503696    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52820038_029c_4693_8905_7756b027c59a.slice/crio-1cca9c62c831cd9c7458f79bf72108ce426f7c720c7da99dbfe7e2d5b20ca083 WatchSource:0}: Error finding container 1cca9c62c831cd9c7458f79bf72108ce426f7c720c7da99dbfe7e2d5b20ca083: Status 404 returned error can't find the container with id 1cca9c62c831cd9c7458f79bf72108ce426f7c720c7da99dbfe7e2d5b20ca083
Nov 02 09:01:55.504317 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:55.504263    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"]
Nov 02 09:01:56.324520 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:56.324477    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq" event={"ID":"52820038-029c-4693-8905-7756b027c59a","Type":"ContainerStarted","Data":"3327ea5cdca37c226b881fcfed1564434c76eeacd8d2fb4c7ed16573d4b09933"}
Nov 02 09:01:56.324968 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:56.324542    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq" event={"ID":"52820038-029c-4693-8905-7756b027c59a","Type":"ContainerStarted","Data":"1cca9c62c831cd9c7458f79bf72108ce426f7c720c7da99dbfe7e2d5b20ca083"}
Nov 02 09:01:57.338507 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:57.338462    4898 generic.go:356] "Generic (PLEG): container finished" podID="52820038-029c-4693-8905-7756b027c59a" containerID="3327ea5cdca37c226b881fcfed1564434c76eeacd8d2fb4c7ed16573d4b09933" exitCode=0
Nov 02 09:01:57.339301 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:57.339269    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq" event={"ID":"52820038-029c-4693-8905-7756b027c59a","Type":"ContainerDied","Data":"3327ea5cdca37c226b881fcfed1564434c76eeacd8d2fb4c7ed16573d4b09933"}
Nov 02 09:01:58.788561 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:58.788107    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:58.920314 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:58.920268    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52820038-029c-4693-8905-7756b027c59a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52820038-029c-4693-8905-7756b027c59a-maas-api-openshift-ca\") pod \"52820038-029c-4693-8905-7756b027c59a\" (UID: \"52820038-029c-4693-8905-7756b027c59a\") "
Nov 02 09:01:58.920560 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:58.920410    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52820038-029c-4693-8905-7756b027c59a-kube-api-access-5w8kv\" (UniqueName: \"kubernetes.io/projected/52820038-029c-4693-8905-7756b027c59a-kube-api-access-5w8kv\") pod \"52820038-029c-4693-8905-7756b027c59a\" (UID: \"52820038-029c-4693-8905-7756b027c59a\") "
Nov 02 09:01:58.920825 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:58.920773    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52820038-029c-4693-8905-7756b027c59a-maas-api-openshift-ca" pod "52820038-029c-4693-8905-7756b027c59a" (UID: "52820038-029c-4693-8905-7756b027c59a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:01:58.921284 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:58.921261    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52820038-029c-4693-8905-7756b027c59a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52820038-029c-4693-8905-7756b027c59a-maas-api-xks-ca\") pod \"52820038-029c-4693-8905-7756b027c59a\" (UID: \"52820038-029c-4693-8905-7756b027c59a\") "
Nov 02 09:01:58.921974 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:58.921944    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52820038-029c-4693-8905-7756b027c59a-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:58.923395 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:58.923349    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52820038-029c-4693-8905-7756b027c59a-kube-api-access-5w8kv" pod "52820038-029c-4693-8905-7756b027c59a" (UID: "52820038-029c-4693-8905-7756b027c59a"). InnerVolumeSpecName "kube-api-access-5w8kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:01:58.923554 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:58.923521    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52820038-029c-4693-8905-7756b027c59a-maas-api-xks-ca" pod "52820038-029c-4693-8905-7756b027c59a" (UID: "52820038-029c-4693-8905-7756b027c59a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:01:59.023689 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:59.023570    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w8kv\" (UniqueName: \"kubernetes.io/projected/52820038-029c-4693-8905-7756b027c59a-kube-api-access-5w8kv\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:59.023689 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:59.023623    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52820038-029c-4693-8905-7756b027c59a-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:01:59.353321 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:59.353230    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq" event={"ID":"52820038-029c-4693-8905-7756b027c59a","Type":"ContainerDied","Data":"1cca9c62c831cd9c7458f79bf72108ce426f7c720c7da99dbfe7e2d5b20ca083"}
Nov 02 09:01:59.353321 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:59.353262    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2898fca-6thlq"
Nov 02 09:01:59.353321 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:01:59.353285    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cca9c62c831cd9c7458f79bf72108ce426f7c720c7da99dbfe7e2d5b20ca083"
Nov 02 09:02:05.310120 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.310084    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"]
Nov 02 09:02:05.310551 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.310536    4898 cpu_manager.go:414] "containerMap: removing container" podUID="52820038-029c-4693-8905-7756b027c59a" containerName="revoke-keys"
Nov 02 09:02:05.310609 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.310552    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52820038-029c-4693-8905-7756b027c59a" containerName="revoke-keys"
Nov 02 09:02:05.310649 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.310644    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="52820038-029c-4693-8905-7756b027c59a" containerName="revoke-keys"
Nov 02 09:02:05.314123 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.314101    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:05.341775 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.341739    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:02:05.342002 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.341931    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:02:05.342091 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.341999    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:02:05.342091 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.342085    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:02:05.345357 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.345322    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"]
Nov 02 09:02:05.489525 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.489486    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5147abfb-7dkll\" (UID: \"51e07c60-2161-42c5-9e79-15cd81b60778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:05.489731 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.489621    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5147abfb-7dkll\" (UID: \"51e07c60-2161-42c5-9e79-15cd81b60778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:05.489823 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.489801    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnlgt\" (UniqueName: \"kubernetes.io/projected/51e07c60-2161-42c5-9e79-15cd81b60778-kube-api-access-vnlgt\") pod \"maas-api-revoke-sub-5147abfb-7dkll\" (UID: \"51e07c60-2161-42c5-9e79-15cd81b60778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:05.590677 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.590567    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnlgt\" (UniqueName: \"kubernetes.io/projected/51e07c60-2161-42c5-9e79-15cd81b60778-kube-api-access-vnlgt\") pod \"maas-api-revoke-sub-5147abfb-7dkll\" (UID: \"51e07c60-2161-42c5-9e79-15cd81b60778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:05.590677 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.590670    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5147abfb-7dkll\" (UID: \"51e07c60-2161-42c5-9e79-15cd81b60778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:05.591082 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.590733    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5147abfb-7dkll\" (UID: \"51e07c60-2161-42c5-9e79-15cd81b60778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:05.591516 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.591485    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5147abfb-7dkll\" (UID: \"51e07c60-2161-42c5-9e79-15cd81b60778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:05.593424 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.593399    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5147abfb-7dkll\" (UID: \"51e07c60-2161-42c5-9e79-15cd81b60778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:05.614453 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.614418    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnlgt\" (UniqueName: \"kubernetes.io/projected/51e07c60-2161-42c5-9e79-15cd81b60778-kube-api-access-vnlgt\") pod \"maas-api-revoke-sub-5147abfb-7dkll\" (UID: \"51e07c60-2161-42c5-9e79-15cd81b60778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:05.628277 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.628240    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:05.937151 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:05.937119    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"]
Nov 02 09:02:05.938216 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:02:05.938176    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e07c60_2161_42c5_9e79_15cd81b60778.slice/crio-48ef349cf4ae22a468addf532826cde4ec5476ed89ddbe2a218965493e39b67b WatchSource:0}: Error finding container 48ef349cf4ae22a468addf532826cde4ec5476ed89ddbe2a218965493e39b67b: Status 404 returned error can't find the container with id 48ef349cf4ae22a468addf532826cde4ec5476ed89ddbe2a218965493e39b67b
Nov 02 09:02:06.399915 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:06.399852    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll" event={"ID":"51e07c60-2161-42c5-9e79-15cd81b60778","Type":"ContainerStarted","Data":"afbdc47b28a3f1b7c760dc577e5e33465335e5fa3831e9a4d787d7c4786ea517"}
Nov 02 09:02:06.400305 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:06.399933    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll" event={"ID":"51e07c60-2161-42c5-9e79-15cd81b60778","Type":"ContainerStarted","Data":"48ef349cf4ae22a468addf532826cde4ec5476ed89ddbe2a218965493e39b67b"}
Nov 02 09:02:06.442138 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:06.442070    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll" podStartSLOduration=1.442051748 podStartE2EDuration="1.442051748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:02:06.440270917 +0000 UTC m=+5877.130597908" watchObservedRunningTime="2026-11-02 09:02:06.442051748 +0000 UTC m=+5877.132378719"
Nov 02 09:02:07.413705 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:07.413662    4898 generic.go:356] "Generic (PLEG): container finished" podID="51e07c60-2161-42c5-9e79-15cd81b60778" containerID="afbdc47b28a3f1b7c760dc577e5e33465335e5fa3831e9a4d787d7c4786ea517" exitCode=0
Nov 02 09:02:07.414262 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:07.413756    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll" event={"ID":"51e07c60-2161-42c5-9e79-15cd81b60778","Type":"ContainerDied","Data":"afbdc47b28a3f1b7c760dc577e5e33465335e5fa3831e9a4d787d7c4786ea517"}
Nov 02 09:02:08.908463 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:08.908070    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:08.947756 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:08.947719    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-xks-ca\") pod \"51e07c60-2161-42c5-9e79-15cd81b60778\" (UID: \"51e07c60-2161-42c5-9e79-15cd81b60778\") "
Nov 02 09:02:08.948000 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:08.947782    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-openshift-ca\") pod \"51e07c60-2161-42c5-9e79-15cd81b60778\" (UID: \"51e07c60-2161-42c5-9e79-15cd81b60778\") "
Nov 02 09:02:08.948000 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:08.947803    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e07c60-2161-42c5-9e79-15cd81b60778-kube-api-access-vnlgt\" (UniqueName: \"kubernetes.io/projected/51e07c60-2161-42c5-9e79-15cd81b60778-kube-api-access-vnlgt\") pod \"51e07c60-2161-42c5-9e79-15cd81b60778\" (UID: \"51e07c60-2161-42c5-9e79-15cd81b60778\") "
Nov 02 09:02:08.948167 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:08.948119    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-openshift-ca" pod "51e07c60-2161-42c5-9e79-15cd81b60778" (UID: "51e07c60-2161-42c5-9e79-15cd81b60778"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:02:08.949963 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:08.949930    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-xks-ca" pod "51e07c60-2161-42c5-9e79-15cd81b60778" (UID: "51e07c60-2161-42c5-9e79-15cd81b60778"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:02:08.950251 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:08.950220    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e07c60-2161-42c5-9e79-15cd81b60778-kube-api-access-vnlgt" pod "51e07c60-2161-42c5-9e79-15cd81b60778" (UID: "51e07c60-2161-42c5-9e79-15cd81b60778"). InnerVolumeSpecName "kube-api-access-vnlgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:02:09.049611 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:09.049570    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:02:09.049611 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:09.049607    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e07c60-2161-42c5-9e79-15cd81b60778-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:02:09.049611 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:09.049617    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnlgt\" (UniqueName: \"kubernetes.io/projected/51e07c60-2161-42c5-9e79-15cd81b60778-kube-api-access-vnlgt\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:02:09.445195 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:09.445163    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll"
Nov 02 09:02:09.445195 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:09.445173    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5147abfb-7dkll" event={"ID":"51e07c60-2161-42c5-9e79-15cd81b60778","Type":"ContainerDied","Data":"48ef349cf4ae22a468addf532826cde4ec5476ed89ddbe2a218965493e39b67b"}
Nov 02 09:02:09.445427 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:09.445231    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48ef349cf4ae22a468addf532826cde4ec5476ed89ddbe2a218965493e39b67b"
Nov 02 09:02:14.989016 ip-10-0-134-79 kubenswrapper[4898]: E1102 09:02:14.988969    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 09:02:15.672425 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.672387    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"]
Nov 02 09:02:15.672992 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.672969    4898 cpu_manager.go:414] "containerMap: removing container" podUID="51e07c60-2161-42c5-9e79-15cd81b60778" containerName="revoke-keys"
Nov 02 09:02:15.673124 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.672992    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e07c60-2161-42c5-9e79-15cd81b60778" containerName="revoke-keys"
Nov 02 09:02:15.673124 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.673107    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e07c60-2161-42c5-9e79-15cd81b60778" containerName="revoke-keys"
Nov 02 09:02:15.676709 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.676678    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:15.709307 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.709261    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"]
Nov 02 09:02:15.711168 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.711141    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlxx\" (UniqueName: \"kubernetes.io/projected/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-kube-api-access-9hlxx\") pod \"maas-api-revoke-sub-a1ddee77-8mk67\" (UID: \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:15.711332 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.711290    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ddee77-8mk67\" (UID: \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:15.711479 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.711399    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ddee77-8mk67\" (UID: \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:15.715262 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.715238    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:02:15.715360 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.715299    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:02:15.726736 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.726700    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:02:15.726736 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.726699    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:02:15.812540 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.812505    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlxx\" (UniqueName: \"kubernetes.io/projected/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-kube-api-access-9hlxx\") pod \"maas-api-revoke-sub-a1ddee77-8mk67\" (UID: \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:15.812744 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.812572    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ddee77-8mk67\" (UID: \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:15.812744 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.812608    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ddee77-8mk67\" (UID: \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:15.813409 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.813386    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ddee77-8mk67\" (UID: \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:15.815191 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.815171    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ddee77-8mk67\" (UID: \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:15.836243 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.836210    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlxx\" (UniqueName: \"kubernetes.io/projected/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-kube-api-access-9hlxx\") pod \"maas-api-revoke-sub-a1ddee77-8mk67\" (UID: \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:15.994480 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:15.994381    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:16.296592 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:16.296555    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"]
Nov 02 09:02:16.297113 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:02:16.297087    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9df1c9a_ccbb_41a9_890d_9e91d9206b2f.slice/crio-471ee05b479ee9e2d0287de21166f3c3bb2a844fd90ff4261fccec624f9abbec WatchSource:0}: Error finding container 471ee05b479ee9e2d0287de21166f3c3bb2a844fd90ff4261fccec624f9abbec: Status 404 returned error can't find the container with id 471ee05b479ee9e2d0287de21166f3c3bb2a844fd90ff4261fccec624f9abbec
Nov 02 09:02:16.491443 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:16.491409    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67" event={"ID":"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f","Type":"ContainerStarted","Data":"471ee05b479ee9e2d0287de21166f3c3bb2a844fd90ff4261fccec624f9abbec"}
Nov 02 09:02:17.502272 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:17.502154    4898 generic.go:356] "Generic (PLEG): container finished" podID="b9df1c9a-ccbb-41a9-890d-9e91d9206b2f" containerID="2176ed095e82a37c23749b5e10c6e3e4bd989bf904dc30cc3cfecd372bd30986" exitCode=0
Nov 02 09:02:17.502846 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:17.502348    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67" event={"ID":"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f","Type":"ContainerDied","Data":"2176ed095e82a37c23749b5e10c6e3e4bd989bf904dc30cc3cfecd372bd30986"}
Nov 02 09:02:18.941284 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:18.941245    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:18.975914 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:18.972454    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-kube-api-access-9hlxx\" (UniqueName: \"kubernetes.io/projected/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-kube-api-access-9hlxx\") pod \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\" (UID: \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\") "
Nov 02 09:02:18.975914 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:18.972533    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-xks-ca\") pod \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\" (UID: \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\") "
Nov 02 09:02:18.975914 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:18.972627    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-openshift-ca\") pod \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\" (UID: \"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f\") "
Nov 02 09:02:18.975914 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:18.973327    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-openshift-ca" pod "b9df1c9a-ccbb-41a9-890d-9e91d9206b2f" (UID: "b9df1c9a-ccbb-41a9-890d-9e91d9206b2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:02:18.990765 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:18.990418    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-kube-api-access-9hlxx" pod "b9df1c9a-ccbb-41a9-890d-9e91d9206b2f" (UID: "b9df1c9a-ccbb-41a9-890d-9e91d9206b2f"). InnerVolumeSpecName "kube-api-access-9hlxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:02:19.018913 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:19.016031    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-xks-ca" pod "b9df1c9a-ccbb-41a9-890d-9e91d9206b2f" (UID: "b9df1c9a-ccbb-41a9-890d-9e91d9206b2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:02:19.086928 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:19.074588    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:02:19.086928 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:19.074633    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hlxx\" (UniqueName: \"kubernetes.io/projected/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-kube-api-access-9hlxx\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:02:19.086928 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:19.074651    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9df1c9a-ccbb-41a9-890d-9e91d9206b2f-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:02:19.540292 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:19.540257    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67" event={"ID":"b9df1c9a-ccbb-41a9-890d-9e91d9206b2f","Type":"ContainerDied","Data":"471ee05b479ee9e2d0287de21166f3c3bb2a844fd90ff4261fccec624f9abbec"}
Nov 02 09:02:19.540468 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:19.540307    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="471ee05b479ee9e2d0287de21166f3c3bb2a844fd90ff4261fccec624f9abbec"
Nov 02 09:02:19.540468 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:19.540362    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ddee77-8mk67"
Nov 02 09:02:26.028168 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.028075    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"]
Nov 02 09:02:26.028596 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.028580    4898 cpu_manager.go:414] "containerMap: removing container" podUID="b9df1c9a-ccbb-41a9-890d-9e91d9206b2f" containerName="revoke-keys"
Nov 02 09:02:26.028650 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.028597    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9df1c9a-ccbb-41a9-890d-9e91d9206b2f" containerName="revoke-keys"
Nov 02 09:02:26.028702 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.028692    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9df1c9a-ccbb-41a9-890d-9e91d9206b2f" containerName="revoke-keys"
Nov 02 09:02:26.031923 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.031880    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:26.050439 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.050401    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"]
Nov 02 09:02:26.058797 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.058759    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:02:26.058992 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.058837    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mcz2m\""
Nov 02 09:02:26.084640 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.084589    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:02:26.084837 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.084650    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:02:26.188352 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.188308    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b702501-wv8s2\" (UID: \"33881a9d-88a4-465a-9c49-5d845d03a68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:26.188352 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.188356    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f79ps\" (UniqueName: \"kubernetes.io/projected/33881a9d-88a4-465a-9c49-5d845d03a68e-kube-api-access-f79ps\") pod \"maas-api-revoke-sub-7b702501-wv8s2\" (UID: \"33881a9d-88a4-465a-9c49-5d845d03a68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:26.188685 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.188470    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b702501-wv8s2\" (UID: \"33881a9d-88a4-465a-9c49-5d845d03a68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:26.289880 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.289777    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b702501-wv8s2\" (UID: \"33881a9d-88a4-465a-9c49-5d845d03a68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:26.290110 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.289925    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b702501-wv8s2\" (UID: \"33881a9d-88a4-465a-9c49-5d845d03a68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:26.290110 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.289950    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f79ps\" (UniqueName: \"kubernetes.io/projected/33881a9d-88a4-465a-9c49-5d845d03a68e-kube-api-access-f79ps\") pod \"maas-api-revoke-sub-7b702501-wv8s2\" (UID: \"33881a9d-88a4-465a-9c49-5d845d03a68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:26.290649 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.290626    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b702501-wv8s2\" (UID: \"33881a9d-88a4-465a-9c49-5d845d03a68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:26.292431 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.292407    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b702501-wv8s2\" (UID: \"33881a9d-88a4-465a-9c49-5d845d03a68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:26.313798 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.313767    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f79ps\" (UniqueName: \"kubernetes.io/projected/33881a9d-88a4-465a-9c49-5d845d03a68e-kube-api-access-f79ps\") pod \"maas-api-revoke-sub-7b702501-wv8s2\" (UID: \"33881a9d-88a4-465a-9c49-5d845d03a68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:26.345646 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.345561    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:26.647369 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:26.647337    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"]
Nov 02 09:02:26.648117 ip-10-0-134-79 kubenswrapper[4898]: W1102 09:02:26.648087    4898 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33881a9d_88a4_465a_9c49_5d845d03a68e.slice/crio-8b3ed291ca74625fa2ca18a2eb00e4d3ab70bb58118c82cf9848f07634532aa6 WatchSource:0}: Error finding container 8b3ed291ca74625fa2ca18a2eb00e4d3ab70bb58118c82cf9848f07634532aa6: Status 404 returned error can't find the container with id 8b3ed291ca74625fa2ca18a2eb00e4d3ab70bb58118c82cf9848f07634532aa6
Nov 02 09:02:27.646771 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:27.646704    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2" event={"ID":"33881a9d-88a4-465a-9c49-5d845d03a68e","Type":"ContainerStarted","Data":"9cea6f4aa7f006b84e934b495e40d41f5b1e6a0188ee72dc2e9f7ce8e35aacf9"}
Nov 02 09:02:27.647475 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:27.647445    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2" event={"ID":"33881a9d-88a4-465a-9c49-5d845d03a68e","Type":"ContainerStarted","Data":"8b3ed291ca74625fa2ca18a2eb00e4d3ab70bb58118c82cf9848f07634532aa6"}
Nov 02 09:02:27.741015 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:27.734270    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2" podStartSLOduration=2.734249059 podStartE2EDuration="2.734249059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:02:27.725415121 +0000 UTC m=+5898.415742113" watchObservedRunningTime="2026-11-02 09:02:27.734249059 +0000 UTC m=+5898.424576025"
Nov 02 09:02:28.654397 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:28.654364    4898 generic.go:356] "Generic (PLEG): container finished" podID="33881a9d-88a4-465a-9c49-5d845d03a68e" containerID="9cea6f4aa7f006b84e934b495e40d41f5b1e6a0188ee72dc2e9f7ce8e35aacf9" exitCode=0
Nov 02 09:02:28.654919 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:28.654445    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2" event={"ID":"33881a9d-88a4-465a-9c49-5d845d03a68e","Type":"ContainerDied","Data":"9cea6f4aa7f006b84e934b495e40d41f5b1e6a0188ee72dc2e9f7ce8e35aacf9"}
Nov 02 09:02:29.802049 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:29.802021    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:29.955685 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:29.955590    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33881a9d-88a4-465a-9c49-5d845d03a68e-kube-api-access-f79ps\" (UniqueName: \"kubernetes.io/projected/33881a9d-88a4-465a-9c49-5d845d03a68e-kube-api-access-f79ps\") pod \"33881a9d-88a4-465a-9c49-5d845d03a68e\" (UID: \"33881a9d-88a4-465a-9c49-5d845d03a68e\") "
Nov 02 09:02:29.955685 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:29.955644    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-openshift-ca\") pod \"33881a9d-88a4-465a-9c49-5d845d03a68e\" (UID: \"33881a9d-88a4-465a-9c49-5d845d03a68e\") "
Nov 02 09:02:29.955982 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:29.955716    4898 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-xks-ca\") pod \"33881a9d-88a4-465a-9c49-5d845d03a68e\" (UID: \"33881a9d-88a4-465a-9c49-5d845d03a68e\") "
Nov 02 09:02:29.956148 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:29.956113    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-openshift-ca" pod "33881a9d-88a4-465a-9c49-5d845d03a68e" (UID: "33881a9d-88a4-465a-9c49-5d845d03a68e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:02:29.957820 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:29.957795    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-xks-ca" pod "33881a9d-88a4-465a-9c49-5d845d03a68e" (UID: "33881a9d-88a4-465a-9c49-5d845d03a68e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:02:29.957961 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:29.957938    4898 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33881a9d-88a4-465a-9c49-5d845d03a68e-kube-api-access-f79ps" pod "33881a9d-88a4-465a-9c49-5d845d03a68e" (UID: "33881a9d-88a4-465a-9c49-5d845d03a68e"). InnerVolumeSpecName "kube-api-access-f79ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:02:30.056591 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:30.056529    4898 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f79ps\" (UniqueName: \"kubernetes.io/projected/33881a9d-88a4-465a-9c49-5d845d03a68e-kube-api-access-f79ps\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:02:30.056591 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:30.056585    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-openshift-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:02:30.056591 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:30.056596    4898 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33881a9d-88a4-465a-9c49-5d845d03a68e-maas-api-xks-ca\") on node \"ip-10-0-134-79.ec2.internal\" DevicePath \"\""
Nov 02 09:02:30.667987 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:30.667955    4898 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2"
Nov 02 09:02:30.668194 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:30.667954    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b702501-wv8s2" event={"ID":"33881a9d-88a4-465a-9c49-5d845d03a68e","Type":"ContainerDied","Data":"8b3ed291ca74625fa2ca18a2eb00e4d3ab70bb58118c82cf9848f07634532aa6"}
Nov 02 09:02:30.668194 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:30.668081    4898 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b3ed291ca74625fa2ca18a2eb00e4d3ab70bb58118c82cf9848f07634532aa6"
Nov 02 09:02:44.990837 ip-10-0-134-79 kubenswrapper[4898]: E1102 09:02:44.990794    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 09:02:48.711952 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:48.711910    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5698f55746-5klb6_37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf/authorino/0.log"
Nov 02 09:02:54.051765 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:54.051719    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"]
Nov 02 09:02:54.070291 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:54.065735    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f52945-kqlnv"]
Nov 02 09:02:55.464224 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:02:55.464181    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b6e7de-1a72-4ce2-b6d9-c6149a69e845" path="/var/lib/kubelet/pods/79b6e7de-1a72-4ce2-b6d9-c6149a69e845/volumes"
Nov 02 09:03:01.685299 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:01.685262    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-rcnpx_7c663e0a-d1b2-4ea4-91d3-fe88764f387d/manager/0.log"
Nov 02 09:03:03.282070 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:03.282040    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0567b209-rhdph_f54ae926-f014-4013-9c9c-9b768ac99559/revoke-keys/0.log"
Nov 02 09:03:03.459103 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:03.459065    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-461e1f5b-pxskt_a6831a27-5834-454c-9edd-bd218a4d9c8b/revoke-keys/0.log"
Nov 02 09:03:03.645227 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:03.645140    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5147abfb-7dkll_51e07c60-2161-42c5-9e79-15cd81b60778/revoke-keys/0.log"
Nov 02 09:03:03.821263 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:03.821227    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-62443a94-t8qh4_57cabc7d-81a8-4408-9ed8-775abfb1e13e/revoke-keys/0.log"
Nov 02 09:03:04.007875 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:04.007845    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-653943bc-mbvxd_ec3fd186-ea0c-4961-ac89-47220cbd9b9c/revoke-keys/0.log"
Nov 02 09:03:04.035316 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:04.035282    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"]
Nov 02 09:03:04.047646 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:04.047615    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0567b209-rhdph"]
Nov 02 09:03:04.196052 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:04.196018    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6e649ed8-qtfvw_3ea2338d-ff9c-4bde-920f-babc159d3a05/revoke-keys/0.log"
Nov 02 09:03:04.376474 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:04.376391    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-759d1499-djg8j_e78457a2-98e2-4db6-a4a1-97fcc4c9fd24/revoke-keys/0.log"
Nov 02 09:03:04.555919 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:04.555863    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7b702501-wv8s2_33881a9d-88a4-465a-9c49-5d845d03a68e/revoke-keys/0.log"
Nov 02 09:03:04.738907 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:04.738849    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a1ddee77-8mk67_b9df1c9a-ccbb-41a9-890d-9e91d9206b2f/revoke-keys/0.log"
Nov 02 09:03:04.915931 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:04.915875    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a2898fca-6thlq_52820038-029c-4693-8905-7756b027c59a/revoke-keys/0.log"
Nov 02 09:03:05.099410 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:05.099328    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c713800e-xg5rh_d20df35b-d387-4a7e-b7ff-0a4cff8ecf4a/revoke-keys/0.log"
Nov 02 09:03:05.286280 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:05.286249    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ce8c45bb-g87bx_a2fcbcba-e847-42e3-a9d7-680aac367b00/revoke-keys/0.log"
Nov 02 09:03:05.463065 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:05.463018    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54ae926-f014-4013-9c9c-9b768ac99559" path="/var/lib/kubelet/pods/f54ae926-f014-4013-9c9c-9b768ac99559/volumes"
Nov 02 09:03:05.470656 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:05.470624    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dd14d128-vmzxc_7b8144d3-2c7f-434a-8b0d-2bab4754388c/revoke-keys/0.log"
Nov 02 09:03:05.646613 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:05.646575    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-eb887bca-rcsj4_bf81ff7f-b3b2-4821-abb5-258ed0bea849/revoke-keys/0.log"
Nov 02 09:03:05.834028 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:05.833948    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ed15ac12-wsc7m_d460dfbb-6535-4108-9c62-23de05999e7a/revoke-keys/0.log"
Nov 02 09:03:06.011356 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:06.011322    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ed246961-b59j7_d0c30fd2-94de-42a0-88bf-5fc275e1fc34/revoke-keys/0.log"
Nov 02 09:03:06.190960 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:06.190932    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f46cee4c-jx54p_1fa1ee89-5eb3-4c8e-8e48-518750220e77/revoke-keys/0.log"
Nov 02 09:03:06.375865 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:06.375828    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f95b0dd8-9skn4_ad797edc-37b9-4558-82ce-c23c46a5def3/revoke-keys/0.log"
Nov 02 09:03:08.448180 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:08.448135    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5698f55746-5klb6_37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf/authorino/0.log"
Nov 02 09:03:08.677763 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:08.677727    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-547hn_eddc764b-21f2-4cd2-b5b3-d331ad9c131d/manager/0.log"
Nov 02 09:03:08.869420 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:08.869333    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-nh2gw_b78e88ff-b93b-48c6-b466-cac6e05256d9/manager/0.log"
Nov 02 09:03:09.264306 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:09.264271    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-w95pb_f96911bb-0980-48ba-8c8d-f97c5e897bac/manager/0.log"
Nov 02 09:03:09.715600 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:09.715567    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pvwn5_a2a4d97a-4060-429a-81e6-dffbfd834a6d/manager/0.log"
Nov 02 09:03:10.648226 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:10.648189    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb_afd78003-f5b8-44eb-91bd-ea259bd3ee02/istio-proxy/0.log"
Nov 02 09:03:10.841777 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:10.841746    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bdrxw_f75d9c81-bd5b-4263-b98d-b155f506c1f3/discovery/0.log"
Nov 02 09:03:12.619402 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:12.619368    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-jxxsm_12755c83-2699-430b-8342-9c06b8273215/istio-proxy/0.log"
Nov 02 09:03:12.816103 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:12.816068    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-m4rk6_a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb/istio-proxy/0.log"
Nov 02 09:03:13.018879 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:13.018849    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pnq5t_dea9d7e7-103a-4e9a-8824-e62a72191cac/istio-proxy/1.log"
Nov 02 09:03:13.417565 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:13.417536    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2_0bd507cc-d1ef-4a8a-92c3-8995091dd0a3/istio-proxy/0.log"
Nov 02 09:03:13.943068 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:13.943034    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dd94bff97-89klj_b93e0f22-01e7-4128-a999-da9b96e029f8/router/0.log"
Nov 02 09:03:14.557757 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:14.557723    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pnq5t_dea9d7e7-103a-4e9a-8824-e62a72191cac/istio-proxy/0.log"
Nov 02 09:03:14.988810 ip-10-0-134-79 kubenswrapper[4898]: E1102 09:03:14.988769    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 09:03:15.265830 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:15.265742    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-2srl5_545d9172-81e3-409a-ae24-163f0646b018/main/0.log"
Nov 02 09:03:15.434677 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:15.434647    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-96rr4_9720edfa-6c5a-4522-af50-30700c3310ce/main/0.log"
Nov 02 09:03:16.207928 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:16.207866    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-s2vrv_0e002ad6-98cb-4c95-b514-ef87cf9fb134/main/0.log"
Nov 02 09:03:16.392113 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:16.392085    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-z96hl_147b04eb-2c14-4316-914e-788b949ccb50/main/0.log"
Nov 02 09:03:26.031088 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:26.031047    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"]
Nov 02 09:03:26.042335 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:26.042298    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed15ac12-wsc7m"]
Nov 02 09:03:27.464038 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:27.464000    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d460dfbb-6535-4108-9c62-23de05999e7a" path="/var/lib/kubelet/pods/d460dfbb-6535-4108-9c62-23de05999e7a/volumes"
Nov 02 09:03:31.129359 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:31.129324    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rs7kb_9bb5657a-bb11-4279-86f6-35118656d990/global-pull-secret-syncer/0.log"
Nov 02 09:03:31.297703 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:31.297676    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-nfdng_f85f18d7-a8b8-4634-af97-eb4e71dc5000/connection-checker/0.log"
Nov 02 09:03:31.417121 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:31.417093    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-w6xgl_437c4525-b041-4931-ba40-99c518193657/connection-checker/0.log"
Nov 02 09:03:31.538534 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:31.538499    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zjjxl_78f8157e-9211-4f25-886e-4a9e9338e730/connection-checker/0.log"
Nov 02 09:03:31.822142 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:31.822056    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9t5tl_4741cb0a-8841-40bb-8cbe-5b4cdd7de7d3/konnectivity-agent/0.log"
Nov 02 09:03:32.108085 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:32.107996    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-79.ec2.internal_04aaad1dd656465cdf1b219bba0dba7b/haproxy/0.log"
Nov 02 09:03:39.724114 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:39.724076    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5698f55746-5klb6_37f5a6f2-03e6-4af2-b4a2-4af35c3e5cbf/authorino/0.log"
Nov 02 09:03:40.359380 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:40.359351    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-547hn_eddc764b-21f2-4cd2-b5b3-d331ad9c131d/manager/0.log"
Nov 02 09:03:40.489328 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:40.489299    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-nh2gw_b78e88ff-b93b-48c6-b466-cac6e05256d9/manager/0.log"
Nov 02 09:03:40.821259 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:40.821219    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-w95pb_f96911bb-0980-48ba-8c8d-f97c5e897bac/manager/0.log"
Nov 02 09:03:41.202646 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:41.202602    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pvwn5_a2a4d97a-4060-429a-81e6-dffbfd834a6d/manager/0.log"
Nov 02 09:03:42.764763 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.764723    4898 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"]
Nov 02 09:03:42.765237 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.765209    4898 cpu_manager.go:414] "containerMap: removing container" podUID="33881a9d-88a4-465a-9c49-5d845d03a68e" containerName="revoke-keys"
Nov 02 09:03:42.765237 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.765223    4898 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33881a9d-88a4-465a-9c49-5d845d03a68e" containerName="revoke-keys"
Nov 02 09:03:42.765334 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.765290    4898 memory_manager.go:365] "RemoveStaleState removing state" podUID="33881a9d-88a4-465a-9c49-5d845d03a68e" containerName="revoke-keys"
Nov 02 09:03:42.768703 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.768679    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:42.790166 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.790135    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"]
Nov 02 09:03:42.872994 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.872964    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-25t77\"/\"openshift-service-ca.crt\""
Nov 02 09:03:42.873182 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.873013    4898 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-25t77\"/\"kube-root-ca.crt\""
Nov 02 09:03:42.873182 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.873160    4898 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-25t77\"/\"default-dockercfg-tsbd8\""
Nov 02 09:03:42.917900 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.917851    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ec75e765-422b-480b-9314-9c1fecba2e97-proc\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:42.918080 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.917930    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec75e765-422b-480b-9314-9c1fecba2e97-sys\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:42.918080 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.917983    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgf2\" (UniqueName: \"kubernetes.io/projected/ec75e765-422b-480b-9314-9c1fecba2e97-kube-api-access-cbgf2\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:42.918170 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.918102    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ec75e765-422b-480b-9314-9c1fecba2e97-podres\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:42.918215 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:42.918197    4898 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec75e765-422b-480b-9314-9c1fecba2e97-lib-modules\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:43.019492 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.019401    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec75e765-422b-480b-9314-9c1fecba2e97-lib-modules\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:43.019492 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.019449    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ec75e765-422b-480b-9314-9c1fecba2e97-proc\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:43.019492 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.019482    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec75e765-422b-480b-9314-9c1fecba2e97-sys\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:43.019812 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.019501    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgf2\" (UniqueName: \"kubernetes.io/projected/ec75e765-422b-480b-9314-9c1fecba2e97-kube-api-access-cbgf2\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:43.019812 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.019548    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec75e765-422b-480b-9314-9c1fecba2e97-lib-modules\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:43.019812 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.019555    4898 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ec75e765-422b-480b-9314-9c1fecba2e97-podres\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:43.019812 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.019591    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec75e765-422b-480b-9314-9c1fecba2e97-sys\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:43.019812 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.019605    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ec75e765-422b-480b-9314-9c1fecba2e97-podres\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:43.019812 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.019591    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ec75e765-422b-480b-9314-9c1fecba2e97-proc\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:43.043497 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.043460    4898 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgf2\" (UniqueName: \"kubernetes.io/projected/ec75e765-422b-480b-9314-9c1fecba2e97-kube-api-access-cbgf2\") pod \"perf-node-gather-daemonset-lgkxr\" (UID: \"ec75e765-422b-480b-9314-9c1fecba2e97\") " pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:43.082141 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.082098    4898 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:43.391491 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.391436    4898 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"]
Nov 02 09:03:43.710062 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:43.710027    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-bbgm5_e3bc3d36-b8b2-409b-8c62-66590a8e3a5c/cluster-monitoring-operator/0.log"
Nov 02 09:03:44.201571 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:44.201531    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr" event={"ID":"ec75e765-422b-480b-9314-9c1fecba2e97","Type":"ContainerStarted","Data":"ae14a4aa4e91317511f0e641aa20ca50c3b319e89e6255394d814e92e06378b7"}
Nov 02 09:03:44.202065 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:44.201595    4898 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr" event={"ID":"ec75e765-422b-480b-9314-9c1fecba2e97","Type":"ContainerStarted","Data":"f04bede8994a7dbedf88560b279814a302d7a250d9c1aec889485961faa9e3ce"}
Nov 02 09:03:44.202065 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:44.201749    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:44.253952 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:44.253900    4898 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr" podStartSLOduration=2.253873769 podStartE2EDuration="2.253873769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:03:44.252093187 +0000 UTC m=+5974.942420151" watchObservedRunningTime="2026-11-02 09:03:44.253873769 +0000 UTC m=+5974.944200732"
Nov 02 09:03:44.416051 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:44.416015    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d96jh_9f08529c-e64e-4af2-8ac8-80ecf5bc2906/node-exporter/0.log"
Nov 02 09:03:44.556683 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:44.556601    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d96jh_9f08529c-e64e-4af2-8ac8-80ecf5bc2906/kube-rbac-proxy/0.log"
Nov 02 09:03:44.695904 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:44.695864    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d96jh_9f08529c-e64e-4af2-8ac8-80ecf5bc2906/init-textfile/0.log"
Nov 02 09:03:44.988453 ip-10-0-134-79 kubenswrapper[4898]: E1102 09:03:44.988413    4898 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vhjhr" containerName="extractor"
Nov 02 09:03:48.363611 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:48.363578    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-rcnpx_7c663e0a-d1b2-4ea4-91d3-fe88764f387d/manager/0.log"
Nov 02 09:03:50.066858 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:50.066824    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-wlj5v_06de30c9-de01-4f63-859a-23df97972617/networking-console-plugin/0.log"
Nov 02 09:03:50.218419 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:50.218383    4898 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-25t77/perf-node-gather-daemonset-lgkxr"
Nov 02 09:03:51.253820 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:51.253790    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/2.log"
Nov 02 09:03:51.258035 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:51.258016    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tqzjh_f4ba5ec9-5277-460a-976e-3b1f154f7623/console-operator/3.log"
Nov 02 09:03:51.456908 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:51.456856    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-rcnpx_7c663e0a-d1b2-4ea4-91d3-fe88764f387d/manager/0.log"
Nov 02 09:03:53.883308 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:53.883258    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-b2tsc_d338565e-a9f0-44e8-ba6c-e0d88c253a10/volume-data-source-validator/0.log"
Nov 02 09:03:54.619797 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:54.619761    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-rcnpx_7c663e0a-d1b2-4ea4-91d3-fe88764f387d/manager/0.log"
Nov 02 09:03:56.657712 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:56.657682    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb_afd78003-f5b8-44eb-91bd-ea259bd3ee02/istio-proxy/0.log"
Nov 02 09:03:56.838447 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:56.838412    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5tv7w_08bc1af6-1c3b-4740-9dc0-fa309e83b78a/dns/0.log"
Nov 02 09:03:56.988734 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:56.988653    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5tv7w_08bc1af6-1c3b-4740-9dc0-fa309e83b78a/kube-rbac-proxy/0.log"
Nov 02 09:03:56.994885 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:56.994862    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb_afd78003-f5b8-44eb-91bd-ea259bd3ee02/istio-proxy/0.log"
Nov 02 09:03:57.054006 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:57.053965    4898 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"]
Nov 02 09:03:57.065532 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:57.065499    4898 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e649ed8-qtfvw"]
Nov 02 09:03:57.234591 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:57.234554    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bdrxw_f75d9c81-bd5b-4263-b98d-b155f506c1f3/discovery/0.log"
Nov 02 09:03:57.464695 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:57.464653    4898 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea2338d-ff9c-4bde-920f-babc159d3a05" path="/var/lib/kubelet/pods/3ea2338d-ff9c-4bde-920f-babc159d3a05/volumes"
Nov 02 09:03:57.599425 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:57.599390    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bdrxw_f75d9c81-bd5b-4263-b98d-b155f506c1f3/discovery/0.log"
Nov 02 09:03:57.635838 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:57.635800    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k4jxw_539550af-7526-447a-b6e3-d268f758aeb1/dns-node-resolver/0.log"
Nov 02 09:03:57.822928 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:57.822828    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-jxxsm_12755c83-2699-430b-8342-9c06b8273215/istio-proxy/0.log"
Nov 02 09:03:57.884121 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:57.884092    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-m4rk6_a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb/istio-proxy/0.log"
Nov 02 09:03:57.944408 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:57.944372    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pnq5t_dea9d7e7-103a-4e9a-8824-e62a72191cac/istio-proxy/0.log"
Nov 02 09:03:57.948693 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:57.948667    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pnq5t_dea9d7e7-103a-4e9a-8824-e62a72191cac/istio-proxy/1.log"
Nov 02 09:03:58.133052 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:58.132959    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2_0bd507cc-d1ef-4a8a-92c3-8995091dd0a3/istio-proxy/0.log"
Nov 02 09:03:58.194664 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:58.194620    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-jxxsm_12755c83-2699-430b-8342-9c06b8273215/istio-proxy/0.log"
Nov 02 09:03:58.262392 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:58.262364    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-m4rk6_a7f8714b-4e0a-4515-bf6a-6d53f8b2cdcb/istio-proxy/0.log"
Nov 02 09:03:58.330452 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:58.330394    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pnq5t_dea9d7e7-103a-4e9a-8824-e62a72191cac/istio-proxy/0.log"
Nov 02 09:03:58.336299 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:58.336275    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pnq5t_dea9d7e7-103a-4e9a-8824-e62a72191cac/istio-proxy/1.log"
Nov 02 09:03:58.336565 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:58.336550    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dd94bff97-89klj_b93e0f22-01e7-4128-a999-da9b96e029f8/router/0.log"
Nov 02 09:03:58.444092 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:58.444059    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-mjtj2_0bd507cc-d1ef-4a8a-92c3-8995091dd0a3/istio-proxy/0.log"
Nov 02 09:03:58.658658 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:58.658627    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dd94bff97-89klj_b93e0f22-01e7-4128-a999-da9b96e029f8/router/0.log"
Nov 02 09:03:59.055738 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:03:59.055705    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hd4ds_a6da9cb8-1262-4288-a7fb-5991fff5c41a/node-ca/0.log"
Nov 02 09:04:01.317731 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:04:01.317694    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f78brcb_afd78003-f5b8-44eb-91bd-ea259bd3ee02/istio-proxy/0.log"
Nov 02 09:04:01.922120 ip-10-0-134-79 kubenswrapper[4898]: I1102 09:04:01.922088    4898 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bdrxw_f75d9c81-bd5b-4263-b98d-b155f506c1f3/discovery/0.log"